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University of Pittsburgh, 1974 

Tliis study was undertaken to test the effectiveness of a computer- 
assisted tutorial, (CAI) program in teaching the novice and transient user 
the procedures for conducting a search on The New York Times Information * 
Bank. It was also hoped that in the course of the experiment the learning 
process of the novice user would be traced to isolate the thesaural from 
mechanical problems. As a very important by-product, the process of this 
study has yielded a detailed analysis of search failures due to mis- 
interpretation of on-line instructional messages. 

Based upon a feasibility study involving thirty five participants from 
the Graduate School of Library and Information Sciences, University of 
Pittsburgh, a two-part experiment was designed to compare the relative 
effectiveness of the CAI program and the printed instructions accompanying 
the system. Sixty four volunteers from the entire academic community 
participated in tliis experiment. 

The quantiLative as well as the qualitative analysis of data clearly 
support the hypothesis that Uu» computer-assisted tutorial program Cor The 
New York Times Information Bank is a more effective training cool than the 
printed instructions offered, by the system. What is also clearly shown by 
the data is that the tutorial program, though more effective than the printed 
instructions, is far from being a ful 'y effective training medium, Tlie rate 
of succefss in completing a meaningful search on The New York Times Information 
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THE NEW YORK TIMES INFORMATION BANK IN AN 
ACADEMIC ENVIRONMENT AND A COMPUTER-ASSISTED 
TUTORIAL FOR ITS NON- SPECIAL 1ST USERS 

♦ 

Dlnch Moghdam, Ph.D. 
University of Pittsburgh, 1974 

Tills study was undertaken to test the effectiveness of a computer- 
assisted tutorial (CAI) program in teaching the novice and transient user 
the procedures for conducting a search on The New York Times Information 
Bank. It was also hoped that in the course of the experiment the learning 
process of the novice user would be traced to isolate the thesaural from 
mechanical problems. As a very important by-product, the process of this 
study has yielded a detailed analysis of search failures due to mis- 
interpretation of on-line instructional messages. 

Based upon a feasibility study involving thirty five participants from 
the Graduate Scliool of Library and Information Sciences, University of 
Pittsburgh, a two-part experiment was designed to compare the relative 
effectiveness of the CAI program and the princcd instructions accompanying 
the system. Sixty four volunteers from the entire acadvmic community 
part Icf pa tod In this exporimeiit. 

Tlie qiianti tatlvo as well as the qualitative analysis of data clearly 
support the hypothesis that L\n\ computi^r-assisted tut5»rlal proj»ram for Tho 
New York Times Information Bank is a more effective training cool than the 
printed instructions offered by the system. What is also clearly shown by 
the data is that the tutorial program, though more effective than tho printed 
Instruuclons, is far from being a fully effective training medium. Hie rate* 
of success in completing a meaningful search on The New York Times Information 
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Hunk without any human aHsiatanco was A7% for thoao who wore exposed to cho 
30-minute CAI session and dropped to 8% for those who had access to printed 
Instnictlons alone. Shortcomings of both methods oC instruction are discussed. 
The author agrees with the majority of systems designers that, In its present 
stage of development, on-line information retrieval systems must offer live 
help to complement other forms of instruction. 

The qualitative findings of the first-time user/system encounter were 
analyj^od to determine the more subtle reasons for search failure, A detailed 
analysis of user reactions to The New York Times Information Bank on-line 
instructional messages is provided and recommendations based upon these 
findings are limited to the bounds of the physical capacity of the present 
system structure. 
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CHAPTER I 

Ihe New York Times Information Bank Is an on-line, 
Interactive InforMtion retrieval system. It has been 
tinder active development since mid-1966, although it was 
thought about and talked about for several years prior 
to that. In another sense, the origins of The Information 
Bank go back to 1851. lhat was the year that the New York 
Times began publishing a newspaper and simultaneously began 
oomplling an index. (1) 

The Information Bank, designed through the Joint effort of IBM and 
Ihe New York Times, became fully operational in February of 1973. It pro- 
vides virtually full coverage of all news articles and features from Ihe 
New York Times as well as selective access to material from about sixty 
other newspapers and periodicals. As of October 1974 there are over 
800,000 citations and 400,000 index terms stored in the data base vMch 
extends back to January 1969. (2^) The processing goal for current mater- 
ial is within four working days after ptiblication of the final Late City 

i ' ■ 

Edition of The New York Times. The estimated input per year under normal 
circumstances is considered to be ^about 100,000 Times articles and 150,000 
non-Times articles." (3^) 

All indexing and abstrac^JJlg is done by subject-specialists from 
clippings assigned to them by a senior editor. The full text of these 
clippings are mlcroflched and made available to system users as a back up 
file for the Information Bank. However, no full text is provided for the 
non-Times sources due to copyright complications. Oc. John Rothman, 



Director of Information Services of The New York Times, describes the under- 
lying Issues leading to the development of the Information Bank as follows: 

Bie Times uuidexrtook this project for two reasons: 
to absorb already existing information services and 
offer a new service to the outside world and thus 
create a new source of revenue. (4) 

As a part of exploring the "outside" market for this information 
system, the Information Bank was installed at the University of Pittsburgh 
on an experimental basis in November of 1972^ At that time this university 
became the first remote user of the InformaUon Bank., and remained the only 
university in the country to have access to this data base until July of 1974. 

As the largest computerized general information retrieval system in 
the world, (5) The New York Times Information Bank offers unparalleled oppor- 
tuniUes for studying various aspects of man-machine ccronunications. The 
file content appeals to a potentially large user population, and the system 
has been designed with the concept that each user can interact with the com- 
puter without the aid of an intermediary. Rothman comments: 

Ihe genius of the Information Bank is its sophistica- 
tion combined with ease of operation. It is possible for 
a relatively untrained person to get the information he 
wants; even the most complex searches can be ccsnpleted 
within minutes by a novice. (6) 

The uniqueness of the opportunity for research with the Information 
Bank was not overlooked by the staff of the Campus-Based Information System 
at the University of Pittsburgh. As the njechanics of the operation of the 
Information Bank were smoothed and a general trend in usage established 
during the first year, more attention was given to the systematic gathering 
of information about the user population and their reactions to the system. 
It was through such observations that the total dependence of the first-time 
users on the Information Bank attendant became noticeable. 



At first glance, this observation seemed to be in total conflict 
•with the aim of the Information Bank as a "self- teaching and self-service" (7) 
system. However, upon closer examination, it became clear that the system had 
confronted a totally new user population for the first time. In this academic 
community one could no longer rely upon a relatively small and permanent group 
of users who intended to refer to the Information Bank as a. part of its daily 
activities. Rather, what was to establish itself as the major user population 
included a very large and transient student group coupled with the casual, 
somewhat curious, and often skeptical librarians, staff, and faculty members. 

Extensive Initial publicity, combined with an "open-door" policy— 
the terminal is situated in the center of a glass-enclosed smoking room on 
the ground floor of the main campus library— led to an overwhelming reception 
by the campus community. The majority of users had little time to devote to 
a thorough training session; and first-time users of the system far exceeded 
the repeat users who did not need full supervision at the terminal. 

As the number of users multiplied, time restrictions had to be placed 
on each individual request to provide for maximum use of the system. In order 
to make the most efi'icient use of time spent at the terminal, two choices were 
available. One was to delegate all searches to a trained operator; the other, 
to train each individual user before he began his own search on-line. 

Statement of the Problem 

In keeping with the objectives of The New York Tiroes Information Bank 
as a self-teaching and self-service system, the concept of search delegation 
was ruled out. Such delegation would have defeated the basic purpose of an 
interactive information retrieval system: that of direct contact between 
the user and the data base. Unus, various means for instructing first-time 



users were considered. These Included group demons tratlons, video-taped 
training sessions 9 a slide show, computer-assisted Instruction, ej\d the use 
of printed gxildes and manuals. Over-the-shoulder Instruction has always been 
available and Is considered by the author to be an Irreplaceable mode of In- 
struction under the current circumstances* 

As a preliminary step essential to the development of any training 
effort, the author compiled a list of situations v4iere the users most often 
asked for assistance. It was observed that problems relating to the mechanics 
of conducting a search could be delineated from those encountered during the 
process of term negotiation. As a first step, it %^s decided to deal with the 
fonner problem. 

To this end, the author developed an interactive tutorial program for 
the Information Bank to be used through the campus computer. In designing the 
program, the basic objective was to familiarize the user with all the steps 
involved in formulating a search strategy. Total time necessary for the com- 
pletion of the tutorial was also of the essence. It had been established that 
most users with real information needs but little interest in the mechanics of 
the system could not comfortably devote more than thirty minutes of their time 
to mastering the use of the terminal. Ihus, a thirty minute program was con- 
sidered to be the longest possible Introductory session. 

Weed for the Study 

As this computer-assisted instruction (CAI) session began to be used 
on a reguleur basis in conjunction with The New York Times Information Bank, 
it became evident that seme controlled tests of its effectiveness would be 
in order. It was this particular need that led to the present study. How- 
ever, it was asst:m)ed from the beginning that the crucial issue to be 



determined was where azxl how the flrst*tlme user faces difficulties during 
his on-line interaction with the Information Bank« Ihis information could 
fora the basis for any training effort, regardless of the medium to be used 
in transmission of the instructions* 

Lx)Jcing forwaxrd to an increasing user population, The Ifew York Times 
needs to reconsider the problems of training transient users; those who will 
want access to the vast pool of information available in the data base, but 
whose needs are not constant and permanent enough to warrant extensive train- 
ing. An increasing nvanber of libraries and academic ccmwunities have expressed 
an interest in using the Information Bank as an integral part of their informa- 
tion services programs. It is evident that there will be a growing need to 
experiment with a variety of training media to find a low-cost and efficient 
means of providing assistance to the novice, transient user$ the system. 

Scope and Limitations 

This study was conducted to test the usefulness of a particular inter- 
active tutorial progr;;»m for The New York Times Information Bank. The CAI pro- 
gram was written for the PEP-IO computer, using the University of Pittsburgh's 
CATAUfST prograimtting language. The attempt to analyze user reactions to a 
computerized data base %d.ll be strictly limited to the specific data base and 
system in question. It zaist be noted that the formal eacperiment ended in 
March 1974; thus, the contents of the data base as wll as both printed and 
on-line instructions reflect the March 1974 status of the system. 

The degree of generalizability may be to the user population of the 
Information Bank in an academic environment. However, it is hoped that cer- 
tain principles evolving from this study will be of value to other user 
groups and for futiire development of similar systems. 



Ihe CAI program was originally written for The New York Times data 
base at its 1973 stage of development, and was later revised to incorporate 
changes in instruction^, messages which appeared on-line prior to March 1974. 

In order to familiarize the reader with the state of the art in user 
training for on-line systems, the next chapter will review the general prin- 
ciples of training as well as specific methods of training considered appro- 
priate for users of on-line information retrieval systems. 
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REVriEV OF RELATED LITERATURE IN 



USER TFJ^&i FOR O^f^LIfJS INFORMATION 
RETRIEVAL SYSTEMS 



The advent of technology has led to the design and implementation 
of sophisticated on-line information retrieval systems which answer the need 
for fast and accurate retrieval of information. These systems are now being 
offered to the public through private (such as The New York Times Information 
Bank), governmental (NASA's RECON), and commercial (SDC's ORBIT) institutions. 
Working with such systems is no longer the exclusive domain of the information 
specialist or the librarian. Many systems are being offered for direct access 
by the ultimate user population: the scientist, engineer, scholar, manager, 
and student. More and more emphasis is placed upon the importance of direct 
use; the need to eliminate or at least lessen the role of the intermediary. 

This trend must by nature lead to the realization by system designers 
of the need for effective user training programs. In the 1967 National 
Colloquium on Information Retrieval, J. M. Cavanagh states that 'Vlthout 
thorough training the user may misuse the system or fail to exploit its 
potential, thereby effectively degrading system performance." (1) P. W. 
Lancaster (2) z^tes training alongside systems design, hardware, emd the 
data base a^ one of the four major factors contributing to the effective-^ 
ness of a search in an on--line system. In a more specific instance, P. D. 
Rae has identified user training as a major obstacle in the use of the SUNY 
system at the Parkinson Information Center. (3^) 
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It Is Burprlslng, theni to witness the general lack of concern 
for education of \isers of information services and systems. (4) Although 
the need for such education had gained national recognition in the early 
1960 's (5), there have been only isolated attempts to insure a full orier>- 
tation program for all users* Lancaster's (6) 1970 review of some of these 
efforts, though not associated directly with on-line systems, provides a 
background on the types of training programs offered in the United States 
and Great Britain for the past ten years. His description of the •niser 
orientation program" for the National Library of Medicine's MEDLARS serves 
as an early example of a comprehensive training session for a computer-based 
information retrieval system. 

At this point, before investigating various training media and facili- 
ties, we must make clear distinction between the goals of ••training" and 
•<edi>ciition". D. H. Holding (2) distinguishes the training concept as an 
effort to learn or teach a given skill, as opposed to education which has 
broader aims. "In fact," he states, "many of the problems of ed\x:ation are 
those concerned with deciding what kind of effect we are trying to produce, 
whereas the problems of training lie in discovering the roost effective means 
of aciiieving specified results*" (8^) Of course, this is not to say that there 
is no overlap between education and training. As Karl Shiith (9) points out, 
the six:cess or failure of any educational (and I would add training) program 
aaust tiltimately be judged by how well it prepares the person for the actual 
task. Principles of training are based upon our present knowledge of learn- 
ing processes and take advantage of information provided by learning theorists. 

It is on this basis that Holding classifies training problems tinder 
three general categories: (a) training devices; (b) motivation; (c) training 
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methods. (10) Another problem which the trainer must deal with is task com- 
plexity. "In some cases no amount of training will overcome the disadvan- 
tacjes of a task which has been designed without regard to human factors." (11) 
This statement by Holding is especially applicable in cases where a retrieval 
system, was originally designed for batch processing and was later converted 
for on-line use. However, in most instances, the trainer has no control over 
this design aspect. ' 

In preser± on-line informaticwi retrieval systems the principal train- 
ing device used is the input unit (teletyr.'e, CRT, etc.) of the systera. All 
other devices used during the tredning process are only adjuncts to the "real" 
equipment. 

User motivation has so far been provided by the user's information 
demands in hig^ work environment. Except for a number of experimental systems 
designed and tested in academic settings, information scientists and system 
designers have worked with defined user populations with strong — though not 
necessarily well-defined — information needsj Although a certain level of 
motivation is essential to efficient learning, there are few cases ythere 
the user does not hold this minimtmi requirement. Holding (12) points out 
that with the exception of school children who may be genuinely unwilling 
to learn, motivational factors surrounding training may not deserve separate 
consideration. 

Bius, of all the problem areas identified, training methods become 
our major concern. In present on-line information retrieval systems, the 
following training methods have been used singly or in combination; 
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(a) printed manuals, guides, and visual displays; 

(b) "over the shoulder" ox personal instruction by 
a specialist; 

(c) audio visual presentations; 

(d) on-line instruction at the terminal. (13) 

Of the four methods menUooed, the printed form of Instruction has proven to 
be indispensable. A comprehensive guide to the system Is essential for refers 
ence purposes. Shorter veursions lend themselves more readily to use by the 
novice or transient populaition. As Lancaster has observed, "Some printed 
material will always be necessary." (14) 

To determine the effectiveness of printed Instructions as the pre- 
ferred medium in transmitting information about on-line systems, we must 
first analyze the nature of the instructional messages. The use of an on- 
line system is a complex task which may be divided into two major areas. 
First, the user must get acquainted with the physical, step by step 
"taechanics" of conducting a search. Next, he must learn the techniques 
of formulating search strategies. Another way of representing this dlch- 
otony is to distinguish "verbal" from "motor" activities. On this issue. 
Holding consnents ". . .what is learned verbally cannot always be translated 
Into action, nor can learned actions always be put into words. In fact, 
people may remember information verbally or bodily, giving different scores 
for 'recall' and 'use'." (15) 

Uiis statement brings into light two points. One is that printed 
instructions alone cannot suffice as the only means of training an individ- 
ual for "motor" activities; the other is that a single method of Instruction 
is not necessarily suitable for every user of the system. Some may be able 
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to translate •Verbal" into •^tor" activities more efficiently than others. 
In describing the Intrex experiment, Marcus, Benenfeld, and Kugel report 
that • .no single instruction method or booklet, no single 'style* of 
presentation, no single compromise between brevity and completeness, seerois 
to satisfy all, or even a majority of users." (16) 

Piablished literature dealing with the effectiveness of •'over the 
shoulder** or personal instruction by a traii^ed searcher in the use of on- 
line information retrieval systems is scant. However, there seems to be 
little argtsnent "that this method is much more effective than relying solely 
upon printed instructions. (17) The drawback, of course, is the cost of 
training ixKlividual users in this manner. 

Ihe next two methods attempt to overcome the cost effectiveness 
problem by technically ^i^eproducing a representative on-line session by means 
of sound-slide shows, films, computer simulations, or other forms of on-line 
instruction. Audiovisual presentations have the advantage of being readily 
available and portable for use in remote locations. However^ they lack the 
necessary interaction between trainer and trainee. Also, we still face the 
problem of transfer of learning from a '•verbal'* or •Visual •• form to the 
"taotor'^ or ••action'* form. Holding refers to the differences among people 
in their verbal facility and concludes that •'leamii>g by doing'^ is superior 
to "learning by saying. •• (18 ) 

This concept is reaffirmed by the Intrex experiment which also 
reported that **users learn l>est by doing •• and also added that ••On-line 
instruction is more effective than off-line instruction and probably 
.sufficient for most users if the system itself is reliable. •• (19) 

Lancaster identifies three types of on-line instrxxrtion, using 
the system's terminal. 

U 



!• Use of the terminal to display a conventional set 
of instructions that could eqtially well be pre* 
sented in conventional printed fonn« 

2» Use of computer-aided instruction (CAI) techniques, 
either to give the user a one-time introduction to 
the system or to lead him by the hand in the conduct 
of an actual search. 

3« Incorporation of explanations of specific convnands 
or system features that the user can call up when 
he needs them. (20 ) 

Another possibility for on-line instruction is a CAI program which is not 
directly connected to the information retrieval system, but rather is used 
in conjunction with it. This program may simulate a remote system on a 
local computer and may be used strictly for instructional purposes. 

At first glance, presenting a set of conventional instmctions on- 
l^jie seems to be a great waste of computer time and storage capacity. And 
yet, this Baethod is roost prevalent a^nong present on-line, information re- 
trieval systems. Apart from the fact that no extra effort is expended 
in rewriting the user manual for oxv-line inclusion, two other points may 
be brought up in favor of this type of instruction. First, the user may 
refresh his memory on the use of a particular command or system feature 
at the moment he needs such J'*formation without havii^ to divert his 
attention to an outside source (such as the user manual or an experienced 
user). Also, as the Intrex experience revealed, "We find that users have 
a strong preference for the OT-line instruction over the off-line manuals 
even when these are identical in content. " (21 ) In this report, Marcus 
and others explzdji this phenomenon on the basis of the user's miscOTception 
that computers are more reliable than manuals as well as his desire to focus 
on one place only. A more convincing reason which may be added to these 
observations is the ••novelty" factor of on-line instruction together with a 
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sense of ••movement*' in actively reading iiistructions >diich are being printed 
on paper or a CRT screeny rather than glancing passively at the same instruct 
tions contained in the printed manu2il« 

The third method of on-*line instruction, incorporation of explema- 
tions of specific commaixls or system fea^^ires9 is basically a modif icaW 
of the first method in that there is minlioum interaction between the user 
and the computer during the training process. For example, in the MEDLINE 
system the user has both the option of receiving conventional iiistxrxictions 
on-line before he starts his search, azki as)cing for assistance during his 
search by choosing from a menu of six possible problem areas. By reporting 
the type of problem he has encountered, the user is in fact requesting to 
see a part of the original instructions which dealt with his problem, ihe 
explanations offered are textual and require no interaction beyond prompt* 
ing the computer to continue or stop the Instnictional messages. No attempt 
is made to test the under standin^j of the user concerning the material which 
%ias presented to him. 

Based on a two yezur study of on-line information systems and their 
impact on the user, Iheodore Wolfe states that "The importaxK» of the tutor^ 
ial sequence cannot be over-stressed. It provides the basis for user^omputer 
interaction and by doing so determines in great part the svccess of potential 
users of the system." (22 ) He suggests the use of c:>mputer"-aided instruction 
to acquaint new users of the system with query formulation processes using a 
sample data base. He further cre^dits a good tutorial sequence for its ability 
to create user self confidence, to increase system efficiency, and tiius its 
marketability. However, as late as 1973, Lancaster states that "Although 
instnjction in on-line searching seems to be an obvious application for CAI, 
and many writers have suggested this approach, comparatively little work on 



13 

O A 
^ t 



this application has so far been conducted^** (23,) niis is iK>t to say that 
the field of computer-assisted instruction has also remained static in the 
past few years^ Katurally, major efforts in designing and testing CAI sys- 
tems have been in educational settings. 

The most comprehensive and well doctimented program dealing with the 
use of CAI in elementary and high school settings was started in 1963 at the 
Institute for Mathematical Studies in the Social Sciences at Sbinford* (24) 
An equally important program concerned with the use of CAI in higher educa- 
tion is the University of Illinois* PLATO system* Both programs have been 
active in seeking new ways to reduce the operating costs of computer-assisted 
instruction* (25 ) 

Although these and numerous other studies (26) have shown to roost 
everyone's satJLsf action that CAI is at least as effective as the traditional 
modes of instnjction, there are few \it^ would argue with G* S« Young's 
assertion: 

One of the problems that we face in designing com- 
puter instructional programs is the fact that we )cnow 
almost nothing about the deep>er psychology of learning 
• • • With or without the computer, we do not tanow the 
real implications, only the logical implications, of 
various choices in the curriculum^ (27 ) 

Again vre become aware of a difference in the problems facing an 
••educator** as opposed to those encountered by a ••trainer**. Althoxigh we 
must not ignore the underlying issues of the psychology of learning, our 
goals are quite clear and vre are indeed concerned with the •^logical 
implications •• of the training program* 

For training users of on-line information retrieval systems w© must 
design a program which will be co«?>etitively effective in its teaching capa- 
bilities, take the least amount of user time and be cost effective. Atkinson 
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ranarks that the evalxiatlon of a CAI program Is • •prlzaarlly« • .an eval«» 
uation of the instructional program and as sxich is basically an evaluation of 
the program designer who is the real teacher in a computer^ssisted instruction 
system." (28)' Thus, for training purposes, if the program designer is also an 
expert searcher and is fully familiar with every aspect of the user/si'stem 
interface, he may be capable of producing a CAI sequence which will be at least 
as effective as his conducting a live session of the same duration. 

CAI has the most promising prospects as a tool for training users of 
on-line information retrieval systems. It serves a double purpose in that it 
conveys instructional messages while preparing the user for the type of inters 
action which, he must get accustomed to in working with an on-line system. 
Ihat is, the form (device) as well as the content is useful to the trainee. 
CAI plays an active part in the teaciilng process as opposed to the passivity 
of printed guides or listening to an instructor. In a sense it forces the 
trainee to keep us with the reading and to practice what he has Just learned. 

The •'hands on" experience, mentioned before as being an important 
element in the learning process, is ever present in CAI interaction. Caruso 
simply states "Interactive programs require a tutorial approach." (29) 
Ultimately, if the CAI were enmeshed with the retrieval system, the ideal 
learning environment would be attained. Robert Reinecke (30) has reported 
on a system based on this concept %^*iich is now operational at the Vision 
Information Center of the Harvard Medical School. Heer and Poyle (31) have 
also described the use of a CAI program in conjunction with a retrieval sys- 
tem. 

From a motivational standpoint Ivan Russell claims that CAI offers ' 
additional incentive through "manipulation of the device, the freshness of 
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th« experience, Intrinsic qualities of the leeumlng content, and xrelnforce- 
nents given by the computer's reactions to the student *s responses.** (32) 

Ihere will always be two types of users In most on-line information 
retrieval systems: the information specialists and the practitioners in 
various fields. The training procedure for members of one group may not 
necessarily be appropriate for the other. Most training programs to date 
have been aimed at the infoznnation specialist who can afford to spend longer 
hours on the training process. 

The e^qjerlence at NASA's Goddard Space Flight Center is notable in 
this respect. Scientists and engineers at the Center were not using their 
on^liM information system (RECON) directly. Pinal analysis rwealed that 
since these individuals did not have frequent need to use the system they 
had not developed the needed proficiency in handling system mechanics. Del 
Frate and Riddle coranent that "At the present time we do not forsee that 
there will be any significant iiKrease in direct user use." (32) And yet 
they end with this note of optimism: "Since dreams die hard, we will continue 
to try to build up a hard core direct user group of new and repeat users." (34 ) 

It is thus ijs?>erative that short and effective training programs be 
developed to satisfy the needs of this growing user population. Experiments 
must be conducted to determine the effectiveness of various training media 
designed specifically for transient user groups. We must plan for the 
future of on-line systems and be prepared for the type of users it will 
attract. !rhis can only be done if we take a fresh look at our present 
training tools and practices. 

By placing the greatest emphasis upm training needs of the user 
population, it has not been the author •s intent to minimize the importance 
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of other features of an Interactive information retrieval system. Options 

related to query formulation as well as result manipulation are also of 

paramount importance to the user* 

A recent study by Thomas H. Martin (35) at the Institute for 

Communication Research of Stanford University explores the user needs 

through a feature analysis of eleven on^-line information retrieval 8ys<* 

terns* The minimal features recoraoended by the systens designers ques«- 

tioned by Martin are used as a basis of evaluating The New York Times 

Information Bank user/system interface* Some of the concluding remarks 

by Martin are appropriate to the ultimate goal of establishing inters 

system compatibility: 

What is needed is not agreement regarding the syntax 
of retrieval leinguages but agreement regarding features 
which all systems should possess* 

If steps are taken now to Insxire that systems ztssp>ond 
to the functional needs of users , and that they share com- 
won features, then in the future it should not be difficult 
to couple them together into a national or international 
resource* ( 36 ) 

One of the primary steps which would insure ''that systems respond 
to the functional needs of the users" is to ascertain that such functional 
needs are fully understood eund taken into account by system designers* One 
of the objectives of the present study is to present those needs which have 
come to light during the training process of first-time users of the 
Information Bank* 
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CHAPTER in 
METHODOLOGY 

In relation to other studies concerned with tiser training for on-line 
aysteins, a number of questions came to mind at the inception of this inquiry: 

1. What are the steps involved In the learning process of the novice 
user of the Information Bank? 

2. What is the most advantageous medium for training transient users 
of this on-line system? Is any one medium better suited than others for this 
purpose, or does the answer lie in a well-balanced combination of several 
nedia? 

3. Given a similarity in the content of instructional messages, does 
the user respond better to CAI programs or to printed instructions? Is there 
a significant difference in the user performance at the InformaUon Bank ter- 
minal between the two groups subjected to these training media? 

4. Does learning the mechanics of the system "guarantee" a success- 
ful search in cases where the needed information is clearly within the range 
of information covered in the data base? In other vwrds, how important is 
the user's familiarity with The New York Times thesaurus and the concepts 
underlying an on-line search strategy? 

5. Are there certain categories of users who will be unable to cope 
with the computerized information retrieval system regardless of the initial 
type of instructions they receive, and vAiy? 
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These and similar iquestlons should be the concern of the system 
designers as well as those Involved In user training for on-«line systems. 
Ihm scope of the present project does not allow for an in-depth study of all 
issues related to the topic of user training* Hcjwever, the following hypo- 
theses encompass a number of the above questions* , 

Hypotheses 

1. OJie computer-assisted tutorial program for The New York Times 
Information Bank, in conjunction with printed instructions, is a more effec- 
tive tool than the printed instructions alone for learning to use the 
Information Bank* 

2. The learning process at the terminal is divided into two separable 
areas: the mechanics of formulating a search strategy, and mastering the use 
of The New York Times thesaurus* 

Assumption 

In addition to the system messages displayed on The New York Tiroes 
Information Bank screen, there is a need for supplemental instructions for 
the first-time users of the system* 

The Tutorial Program 

A computer-assisted instructional program was designed to simulate 
the Information Bank search process* Ihe user was asked to conduct a search 
on ••the relationship between automobiles and air pollution in 1972 *'• This 
was considered to be a typical reqxiest from the Information Bank* It was a 
topical it«n requiring the use of two terms (••automobiles'* and ••air pollution ••), 
one modifier (year: 1972), and a Boolean operator (AND)* The program allows 
for the interaction the user would normally encounter in a real search on the 
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mformatlon Bank as well as additional explanatory comments when the user 
fails to make the expected responses • A sample interaction with the program 
is Included in Appendix B% 

Methodology 

Ihe experimental design called for testing the effectiveness of the 
tutorial program by the rate of success the participants displayed in con- 
ducting an actual search on The New York Times Information Bank. A compare 
able group of participants was restricted to using printed instructions as 
the only means of getting acqtaainted with the system. Their performance on 
the Information Bank was used as the control measure against which the tutor- 
ial group was Judged. A feasibility study was carried out to confirm the 
soundness of the design. Minor cheuiges were made in the formal experiment 
based upon the findings of this early study. 

In order to further assure uniformity in res\ilts axxl allow for a 
fair comparison of the two groups , the following precaution2u:y meastires were 
taken: 

1. The final analysis of data was strictly limited to those obtained 
from first-time users of the system >^ had had no prior exposure to the 
Information Bank. 

2. Questions to be answered on-Line as a part of the feasibility 
study were designed to test the full capabilities of the system. Peurticipa- 
tlon in the formal experiment required a screening of the user's information 
needs. Only those individuals whose query seemed to reflect the data base 
contents and full use of system capabilities v^ere asked to participate in the 
experlDnent. 

34 



23 



3« UhnecessAry Interaction between the experimenter and the partlcl- 
pant did not Interrupt the flow of the seeurch process* Participants had been 
asstared that a parallel search would be conducted for them If they were unable 
to obtain expected results. They were also Informed that all of their ques- 
tions concerning the system would be answered immediately upon completion of 
the experiment* 

4m The amount of verbal information offered each participant was 
kept constant in all cases. Whenever the participant insisted upon an ans- 
wer: to a question which would affect the outcome of his search strategy, the 
experiment was considered "terminated" at that point in the search when the 
U£er*s first question was answered. 

5. It was not the intent of this study to meastire the participants* 
degree of frustration. In cases where visible signs of over«-anxiety or frus- 
tration interfered wi^h the normal search process the experimenter gave verbal 
assurance that the participant was not obligated to complete the full search. 
It was assiimed that si>ch cases would have terminated the search under non- 
experimental conditions. As the object of the experiment was to replicate 
the real-life sitxaatlcn, it was deemed unnecessary' to apply any pressxires to 
the participants which they would not have normally encountered in a visit to 
the Information Bank. 

Data Collection and Measurement 

Other than obtaining the printout of interaction with the tutorial pro- 
gram, all data collection was based upon direct observation and questioning of 
the participants. Extensive notes were taken during the user interaction with 
the Information Bank, reflecting every step of the search as well as all com- 
ments offered by the participant and his physical behavior during the search 
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process. Thusy the major findings of the study reflect the experimenter's 
personal view of the user/system Interplay. 

On a more objective level, the following data were obtained for 
each participant In the formal experiment: 

A. Personal characteristics and background information. 

B. Ibtal time spent on training (tutorial and printed instructions). 

C. Total time spent on searching the Information Bank. 

0. The step at i^ch the first significant error was made by the 
participant in an on-line response to an instructional message. 

E. Ihe last step successfully completed on the Information Bank 
terminal. 

Although a quantitative analysis of the data will yield concrete 
evidence as to the effectiveness of the tutorial program, it has been the 
author's Intent to use the qualitative findings of the experiment to ana- 
lyze the more subtle reasons for search failures. Ihe author finds it more 
Important to follow the user/s trend of thought leading to a misinterpreta- 
tion of an instructional message than to simply record—and report—an curror 
when it occxirs. Although it is possible to provide quantitative results in 
relation to an occurence of a certain error, as wdULl be shown in chapters 5, 
6, and 7 there are often a variety of reasons behind any single form of error. 

The New York Times Information Bank computer may be programmed to sys- 
tematically record all Incorrect responses to instructional messages, but what 
will be lacking from such a record is the reason for the user's failure to 
respond correctly. The major contribution of this study, though be it sub- 
jective, is to provide the system designer with a glimpse of the user's 
thou5^t pattern as he first encounters an on-line bibliographic information 
retrieval system. 
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Definition of Terms 

The reader Is referred to Appendix A, the User Guide to the Ihfor^ 
nation Bank for a glossary of terms used In connection with the system. 
Ihe following terms or phrases have been used with a special meaning 
throughout this study. 

•^lue Card" refers to the one-page printed Instructions accompanying the 
tutorial programy which was also made available to the non«* 
tutorial group, it gives a brief explanation of the stages of 
a search axid provides the user with a list of 'hinlversal options" 
or commands available on the Information Bank. A copy of this 
card appeeurs In Appendix B. 

"Non- tutorial Group" refers to the control group of participants who 
received only the printed forms of instruction prior to con- 
ducting an on-line search. 

"Normal Search" refers to %rtiat has been established as a typical search on 
the Information Bank in this academic community. It consists of 
the logical combination of two terms %^ch have already been re- 
stricted by a single modifier. The modification is typically a 
specified date or date range , and the Boolean connector is an AND. 

"Search Failure" applies to the failure of the user to obtain the desired 
results as defined by a "Successful search". Its various causes 
are described in chapter 5. 

"Significant Error" as applied to an error occuring during the course of an 
on-line search refers to an error which is not detected by the user 
prior to its entry into the system. It is an error which alters the 
sequence of search steps, and thus directly affects the search results 
or the user*s perception of the search process. 
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^Succusfixl Search** refers to a search which has yielded positive and com- 
plete results as Judged by the participant. For the purposes of 
this experiment, the participant Is the final Judge on the useful«- 
ness of the obtained documents • 

**lUtorlal Group" refers to the participants who were requested to complete 
the CAI training program before conducting an on-line search on the 
Information Baji)c« 

•TPypical Search**; see: *1farmal Search**. 
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CHAPTER IV 
CONDUCT OP IHE EXPERIMENT 



A total of 99 individuals participated in the experiment to teat 
the effectiveness of the CAI program for The New York Times Information 
Ban)c« The fornal experiment was preceded by a feasibility study which was 
carried out as follows. 

A group of volunteers from the Mechanised Information Retrieval 
course offered at the Graduate School of Library axxl Information Sciences 
%ias randomly divided into two groups on the basis of a pre»set appointment 
schedule* That is, once all appointment hours were filled , every other par^ 
ticipant was assigned to the tutorial group at the moment he arrived for the 
e39)eriment» Ihus, no participant had prior knowledge of his group assignment, 
and the experimenter had no control over the sequence of appointments made* 

Each group was given the same basic information about The New York 
Times Information Bank and the general nianner in which the experiment would 
be condticted. Each of the subjects in one group was required to take the 
interactive tutorial program before proceeding with his search* Individuals 
In the second group %«ere directed to the Information Bank terminal without 
receiving any further instructions* 

At the terminal, both groups %fere given a basic orientation to the 
equipment* The user guides, thesaurus, and other printed instructions fur^ 
nished by The New York Times were available for use by all subjects* Ques- 
tions to be searched were controlled by the experimenter so that involvement 
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In tmm negotiation could be kept constant and at a mlnlmm* These ques« 
tlons were written on Individual cards and selected at random by the usert 
No question was used more than once for each group* The purpose for this 
control was to discourage the participants from discussing the search topic 
with other participating members of the class* 

Because of the objections, raised to some of the test questions as 
being Irrelevant to the needs of the participants, thus thwarting motivation, 
the formal experiment relied upon inquiries of genuine Interest brought by 
each participant* 

The formal experiment was further divided into two sections* Section 
Z was a repetition of the feasibility study, using volunteers from the entire 
academic connrunity, and conducting the search on a topic of personal interest 
to the participant. Assistance in term selection was offered prior to the on« 
line search to offset any difficulties the user may have faced in translating 
his inquiry into •Tcey terms" acceptable to the systan* 

Section II differed from Section I in only one respect* Ko off«-line 
term negotiation %ms offered by the experimenter. The purpose for this single 
variation was to evaluate the effect of familiarity with The New York Times 
thesaxunis on final search restilts. It %iras asstsned that if such familiarity 
is essential to the conduct of a successful search, the success rate would 
drop significantly for Section H* 

Analysis of data and results of the study will be discussed in 
Chapters 5 and 6; followed by an analysis of the instructional media in 
chapters 7 and 8« 



29 

40 



CHAPTER V 

TOE EXPERIMENT FEASIBILITY STUDY; PRELIMINARY DATA ANALYSIS 

To test the feasibility of the planned experiment^ a group of 
students from the Graduate School of Library and Information Sciences was 
recruited to participate in the evaluation project for the first tutorial 
progran* These students were all enrolled in a course on mechanized infor-» 
nation retrieval offered at the Master's and Ph.D. level. 

A total of 46 appointments were set upy four of which were cancelled 
due to other coasaitznents by the participants. Of the remaining 42 9 the fol«- 
lo%iing cases were eliminated from the fix^l analysis: 

System's mechanical failure: 2 cases 

Participants ran out of time or did not concentrate 

properly on the experiment due to time pressures: 2 cases 
niose randomly selected for the control group who had previ- 
ously used the Information Bank: 3 cases 

The quantitative resxilts of this experiment are reported in tables 5.1 
through 5.9. The control group is referred to as '^on-tutorial" throughout 
this report. 

The participants in this experiment were predominantly female 9 re- 
flecting the overall enrollment pattern in the Graduate School of Library 
and Information Sciences. The majority of the participants were worlcing 
toward their first graduate degree and only eight of the thirty-five had 
had any previous e3q>erience with computers. It may be noted that the 
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random selection process placed five of these eight Into the non-tutorlal 
group. Three of these five had used computers In a time-shared environment. 

Except for two Individuals with a science background—one In mathe- 
matlcs and the other In blology~the rest had degrees In the social sciences 
and humanities, with a majority In English literature and education. Well 
over half of the participants had no familiarity with the printed Index of 
The New York Times. 

The total time spent in completing the tutorial program is noted In 
table 5.2. No similar figiares are available for the amount of time taken by 
the non- tutorial group to read the printed instructions and manuals. It was 
especially difficult to obtain accxjtrate measures for the non-tutorial group 
as they had been instructed to read the short handout (see Appendix B) in 
advance of their appointment. Although they were given the opportunity to 
read or review the guides upon arrival , roost participants expressed a desire 
to start the search at once and refer to the manual during the course of the 
search rather than take the time to read all instrvictions before starting. 
Thus, the total time reported for the non- tutorial group in table 5.3 re- 
flects the time taken to review the manual as well as time taken in inters 
action with the computer. 

Of the nineteen participants who took the tutorial, twelve completed 
the program in thirty minutes or less. In this groxjp of twelve, eight had 
fully successful searches on Ihe New York Times Information Bank, while the 
success rate for the rest of this group (those taking longer than thirty 
minutes on the tutorial) was one in the remaining seven cases. This figure 
compares well with the zero success rate fteported for the non-tutorial group, 
(see table 5.4). In looking strictly at the participants' abilities to follow 

31 



through the "mechanical" steps involved in a normal search on the Information 
Bankf we find that the success rate is 12 out of 19 for the tutorial group 
and 3 out of 16 for the non-tutorial group* 

Before concentrating on the specific errors made by each user and 
their significance on the search results, we must consider the general cate«> 
gories of errors which commonly lead to search failure* 

In an in-depth analysis of searches conducted under experimental con- 
ditions as %fell as drawing upon extensive )cnowledge gained while assisting in 
the execution of over two thousemd searches during a two-year period, fotir 
separate and distinguishable types of errors have been isolated: 

!• Conceptual errors 

2« Ihesaural errors 

3« Mechanical errors 

Am Interpretation of on-line instructional messages 

Of course, this does not mean that there are no overlaps between 
these groups* In foctf most comition errors are caused by an intermingling of 
two or more types of individually recognizable errors* Ihese cases will be 
discussed once the basic boundaries of each individual type of error has been 
established and e3q>lored,^^ 
1» Cdnceptual Errors 

These errors are of two basic types: those dealing with pre-conceived 
notions about the capabilities of computers in general, and those directly in- 
volved with the user's concept of The New Yorlc Times Information Bank* 

The pre-conceived notions themselves can be divided into two major 
classes of favorable and unfavorable origins* The computer is seen by some 
only as a ••monster"; an inhumane, ruthless and menacing machine which can 
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cause nothing but trouble • On the other extreme are those irtK> believe that 
"If it is a ccxnputer, it can do anything, ansvrer any question*^ It is often 
quite essential to know about these preconceptions as they color the entire 
search process, even though it may be the user's first actual experience 
with the computer. 

On a more specific level, conceptual problems relating to The New 
York Times Information Bank fall into the following categories: 

A« Some user5/ do not realize the inevitable sequentiality of the 
steps involved in any given search* They do not grasp the idea that each 
step has its own unique set of instructions and that an acceptable response 
to a given step is essential before the computer can take them to the next 
step* 

B* As an extension of the above, some users feel that the computer 
will not accept a message from them unless it is the •'right" choice for that 
step, and that it will not offer an option unless it is applicable to the 
search at hand* niis feeling is reinforced by the fact that they nay have 
encountered an error message from the computer on the first few steps and 
decided that whenever they make an error— though it be Judgemental rather 
than mechanical~the computer will inform them of the error* 

C* There is edso the concept of 'Hinrelatedness" between the in« 
structional messages and the on-line thesaurus* For example, at the point 
of term entry or term selection the computer does not analyze the index 
terms chosen by the user and give instructions according to the type of 
information requested* When the system message relates "Try an inversion 
of this term",* this does not necessarily mean that the inverted form of 

•In order to preserve the integrity of messages appearing "on-line," 
punctuation marks which are not a part of system messages appear outside 
quotation marks throughout this report* 
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this tern is in fact a legal descriptor* Likewise , if the user chooses the 
subject of "economic conditions and trends" and then proceeds to modify this 
term by "sketch" or "byline", the ccinputer is not equipped to respond at once 
to an error of this type. But the user assumes that if the option was made 
available to him, then it must be "legal" and applicable to his search. He 
Bay also assiime that the computer "understands" the concept of his question 
rather than simply responds to the terms used. 

0« Another instemce of a conceptual problem is when the user feels 
that there is an unknown element ahead of him. He may state his problem this 
way: "You never know what the next step is going to be, so how am I expected 
to proceed vdth my inquiry?" An excellent example of this is the user who 
types his date modifier at the point of term entry. He is not sure that later 
in the search there %d.ll be a specific step which will ask him for date modi- 
fication. This individual expresses surprise when an experienced searcher 
moves rapidly from one step to the next without seemingly reading the in- 
structional messages. He asks: "But how ceui you tell what the next step 
is going to be?" Unfortunately, printed instructions such as the User Guide 
to the system do not fully satisfy the information needs of this user. 

E« A totally different set of conceptual problems occur in dealing* 
with the logical connection between terms. This problem presents itself at 
two separate and distinct stages in a seeuxh. First, the user may staxrb his 
search with the idea that his terms vd.ll be automatically connected at the 
point of term entry. Some even go as far as stating their query in terms of 
Boolean logic at this point. Here, the user becomes totally confused when he 
encounters only one of his terms (the last one he typ)ed) on the first term 
selection screen. He asks, '*But what happened to my other terms?" Or he 
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selects the option to *^rlew continuation" on the first tern, assviming that 
somewhere in the thesaiiral display he will encounter the logical combination 
of all his tenns* 

Ihe same conceptual problem reappears , of course , at the logic stage. 
At this point the user may shew his misconception about the logical AND and 
OR. He %^11 usually use the Boolean AND connective when he really means to 
use OR* 

P. As with the logical connectives , there are also other conceptual 
problems related to specific stages of a search. Those dealing with term 
entry have already been stated. Problems with modifiers are less common 
but nonetheless apparent, especially in cases where the user feels that he 
should be able to modify his search at the point of term entry. Many users 
insist upon entering a date {such as 1968, or 1973) as a separate "term." 
They may even go as far as selecting such a "term" as it appears on the 
screen and conceptually lin)clng it with their own search. A contnon example 
is the case of the various congressional sessions which are represented by 
year in the on-line thesaurus. An Individual doing a search on "education 
and schools" will automatically accept the term "1973 session" as one dealing 
with the academic year or school session, assuming that the computer had rec- 
ognized the search as being education-related. 

As stated earlier, it is very difficult to differentiate between 
purely conceptual types of errors and those which deal with definitions of the 
instruction, the mechanics, and the thesaurus-related portions of the search. 
Quite often what appears on the surface to be a ••mechanical" or "thesaural" 
difficulty turns out to be a conceptual misunderstanding on the part of the 
user. 
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2* Ihesaural Errors 

Ihese errors can also have many variations: 

A* In a controlled vocabulary systoi there is always the problem of 
relating the tenns used by the non--specialist to that of the system thesaurus 
Many of the search failures ioce due to a laclc of umderstanding of %rtiat terms 
are acceptable to the system axKl hew certain questions may be manipulated to 
fit the system vocabulary and produce the desired output. As an example we 
may cite the case of an individual needing some references on the attitudes 
of Americans on a topical issue (such as race^ politics or religion) • Ihe 
first-time user is quick to discover that the term "attittades" is not accept- 
able to the system* Even after many tries he may not discover the term 
••public opinion" or "Gallup Poll" as aOLtematives which may serve his purpose 

Also in the thesaurus-related area, but on a somewhat more conceptual 
level, we must address the problems relating to the manror in which appropri- 
ate terms must be chosen for querying the Information Bank. It is difficult 
to convince some users that they must choose "concrete" or "objective" terms 
rather than "subjective" or soft, generic terms* It is hard for some users 
to understand that the terms "major issues ", "potential", 'Valuable" or 
••projected" are unacceptable to the system* 

B* Other than difficulties inherent in the use of systems with con- 
trolled vocabularies, there are certain idiosyncrasies specifically related 
to the on-line thesaurus of Ihe New York Times* Many individuals have asked 
«How do you get material about California (or Nepal, or New York City) which 
is not about its geography?" Ihe problem being that the modifier ^KjEO" 
attached to the term is interpreted as the "geography" of the area rather 
than its "geographic" location* A number of the experimental searches 
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failed becaiise of this slnnple misinterpretation of material presented 
through the on-line thesaurus. 

Similar problems are encountered In loolclng at cross-references 
tagged with obscure abbreviations which are not defined or even spelled out 
on-line. These abbreviations (such as SAG and SA0) aire 9 however , defined as 
part of the glossary on page 30 of the User Guide . 

C« Single Indexing errors , to be expected in such a large and active 
vocabulary file, also lead many first-time users astray. Inconsistent uses 
of hyphenation or inversion, to name two convnon instances^ may separate the 
articles dealing with a particular topic into two or more files which do not 
necessarily follow each other alphabetically and are thus lost to the unini- 
tiated. 

D. An area which may be classified as having both conceptual and 
mechanical elements is the right truncation feature of the on-line thesaxirus. 
TWO separate issues are involved here. First, many users are r>ot convinced 
that the truncation feature actually works. That is, they Insist upon enter- 
ing the full form of the term in (question once they have received a message 
that the truncated form of their term is •'not in the file". This is basic- 
ally a mechanical duplication of effort and by no means serious; it only adds 
to the on-line search time. The next issue, however, can affect the entire 
search strategy. Once the uninitiated user "enters" a term into the system 
and is confronted with a screenful of terms (beginning with the last terra he 
had typed), his immediate interpretation is that all of these terms are some-- 
how related to his search. He may not visualize the list as being strictly 
alphabetical. He is either annoyed at receiving ••unwanted" items, or over- 
Joyed by. the thought that the computer has found not only the one term he had 
asked for but a lot of related terms he did not know existed in the file. 
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The "connective" concept described under above is also visualized 
here. If the user had originally entered two or more terms into the system, 
as soon as he sees the first screen offered for term selection he starts look- 
ing down the list to find the entry which has combined all of his terms • Ihis 
\iser may ask to 'View continuation" through numerous screens in search of that 
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elusive descriptor which fully defines his search topic. Here is the case of 
a conceptual problem affecting term selection. In the process of looking for 
that "ideal" term, the user invariably bypasses the single term which he should 
have picked from the first screen. 

E. Ihe number of postings for each on-line thesaurus terra is given in 
a binary range opposite that entry. There are few, if any, novice users v*io 
pay attention to this very significant piece of information. However, even 
when the user's attention is drawn to the number of citations, his intuitive 
reaction is to stay away from large files. This type of judgement is disas- 
terous for most searches using the AND logic, the roost common search strategy 
on the Information Bank. Ihe user states "But I don't want to see 4000 cita- 
tions; especially when I am not even sure that this is the right file." Again, 
he has conceptually lost sight of the fact that a logical AND intersects two 
files and drastically reduces the output and that he must start vdth a large 
enough file if he expects a reasonable amount of output. 

P. Automatic showing of cross references to particular index terms 
as well as automatic switching from an acronym or syncmym to the^ legal des- 
criptor can also become a source of great confusion to the novice user. 

Suggestions for alleviating some of the above problems appear in 
chapter 8. 
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3. Mechemlcal Errors 

As with the coxiceptual Issues Involved In a search^ there are also 
two basic types of mechanical problems. First, there are those individuals 
who are qxiick to admit that they have no affinity for mechanical objects. 

"I have trouble operating a can opener," offered one participant as she 

I 

approached the terminal. She also e^ressed a fear qxiite prevalent among 
first-time users; the fear for '^breaking the machine" or somehow hurting it 
by pressing the wrong keys. •*! don't like machines" is a rather common ex— 
pression for those who show visible signs of nervousness at the terminal. 
An "inherent mistrust of con?)uters" is stated as the cause for anxiety by 
others. Dealing with individuals of this type may in itself be a valid 
subject for in-<iepth study; it is, however, beyond the capabilities of the 
search assistant to an2tlyze such behavior and prescribe remedies. At best, 
he may ask the individual to stand by while he condxicts a search and explains 
the basics of the search procedure to the patron. 

Here, we are concerned with another type of user group; those who have 
no oven^lming fear of the terminal and etre at least receptive to the idea of 
trying a search on their own. This type of a user may face any one or combina- 
tion of the following mechanical problems; 

A. Purely mechanical errors may be as simple as missing the letter 
"A" on the keyboard and pressing the shift key which is immediately to its 
left. On an Incoterm keyboard such action turns on a red li^t under the 
shift key. The novice user is immediately alarmed at having done some irrep- 
arable damage. Few are able to recognize the problem for what it is and solve 
this minor mystery without the aid of an attendant. 
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Of far greater concem are the various typographical errors made; 
one of the most COTmon being the use of the small letter "t" for the numeral 
This is more evident with those viho are experienced typists. Ihus it 
is even more frustrating for them to get "incorrect" messages from the com- 
puter regarding their typing skills. Also, because the computer's response 
to typographical errors is simply "incorrect response to message. Please 
re-read instructions", the user may misinterpret the message and assume that 
he should have used a different option, format, or terra number rather \1ian 
what he had used. 

TWO factors aggravate this problem. First, many novice users have 
never worked with a touch keyboard. They are not aweure that they may be 
accidentally printing a letter twice by gently tapping on the key. Second, 
once the error message appears on the screen, the user's original message 
has been erased. Thus he no longer has access to what had triggered this 
computer response. Ttiis is very crucied since in most instances of typo- 
graphical errors the user is totally unaware of his mistake. For example, 
if he had unwittingly typed "bb//l" while he had thought of typing "b//l" 
and received an error message, he would assume that he should have used a 
different option sxich as "A//1", or a different format such as "bl//". Or 
he may even conclude that he should have chosen another term number. This 
user is now a prime candidate for becoming conceptually confused due to 
a sii^Dle mechanical failure. 

B. Some mechanical problems contain an inherent conceptual factor. 
For exan?>le, pressing the "Enter" key to relay a message to the computer 
carries with it many xmidentified images in the user's mind as to the real 
meaning of this mechanical act. These images determine his behavior at the 
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texninal* Ihey will determine hen long he Is willing to wait for a response 
and what he will CiJ with his time while waiting. An tmexpected delay usually 
leaves the user anxious and uncomfortable* He may start pressing the ''Enter*' 
key over and crrer again or he may retype his message— though the ::eyboard is 
locked and will not register his typing efforts. If he happens to know about 
the "reset" and "erase" keys on the keyboard, he may reset the screen and type 
in another message on the assumption that it was his original message that had 
caused the delay. All of this, of course, is time and effort wasted by the 
user. It will leave an unfair impression in his mind and color his Judgement 
of the entire system. 

C. A separate set of mechanical errors are those caused by misinter-> 
pretation of on-line instructional messages. Ihese errors, categorized by 
their cause rather than effect, will be dealt with in the next section. 

Barring unusual circumstances, pure mechanical errors do not cause 
search failures. It is usueilly a deeper problem with *»raechanical symptoms" 
%rtilch forces an individual to abandon a search. It is not unusual, however, 
for a person with low tolerance, to discontinue a search because of receiving 
repeated err^r messages due to purely mechzmical errors. 
4. Interpretation of Instructions 

A. It is quite likely that the uninitiated user may find a totally 
wrong, yet Justifiable, definition for phrases such as "system messages", 
"view continuation", "Quit»A", and the like. These messages appear on speci- 
fic screens of The New York Times Information Bank and as stich will be dis- 
cussed in detail in chapter 8. In order to avoid a duplication of such 
discussion at this point, the reader is referred to that chapter for further 
information on search failures due to the misinterpretation of instructional 
messages. 52 
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B« Quite apart from the misinterpretation of instructions, there are 
those cases where the user is unable or unwilling to decipher or interpret the 
instructional messages. Rather than chancing misinterpretation, the user sim« 
ply sits staring at the instrttctions and avoids making any decisions. Itiis 
happens most frequently at the initial stage of a search. Phrases such as 
"I don't understand. . "what does it mean. . "it makes no sense. . ." 
and the like are more common at the identification stage and system message 
screens than at any other point in the search. 

It may be argued that we are really confronted with conceptual prob- 
lems at this point, but it is difficult to s^Jlbst^mtiate this case. In most 
instances if the user is pressed for a reason behind his statement, he will 
refer to the specific instruction at haxxl rather than admit (if that is in 
fact the case} to a broader conceptual confusion. 

Instmctlonal messages are the most common cause for errors commited 
on-line, but it bears emphasis that the cause of search failures cannot al- 
ways be pinpointed to a particular type of error. It is much more likely 
that a combination of all four types of errors lead to an tinsuccessful search. 
While table 5.5 identifies the dominant type of error in a given seeuxh, an 
in-depth analysis reveals a variety of interrelationships. 

In the case of conceptual errors, ten of the fourteen cases led 
directly to other recognizable problems. Seven cases showed immense diffi- 
cxilties in performing the mechanical functions of the search^ v^le three 
cases led to inadequacies in dealing with the thesaurus. The results also 
show the positive effect of the tutorial in acquainting the novice user with 
the underlying concept of an on-line search of The New York Times. 
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However 9 the tutorial group f somewhat relieved of coping with the 
search concept %iaB strongly affected by thesaurus difficulties. In the 
majority of cases , conceptual difficulties took precedence over thesaural 
and mechanical problems; vihile on the next level of performance , mechanics 
of the search had to be mastered before the user showed ax^ evidence of get- 
^tlng involved with the thesaural question. Thus, those individuals showing 
the greatest involvement (and hai;:dshlp) with the thesaurus had had a better 
grasp of the conceptual and mechanical parts of a search. Ihe failure of the 
tutorial program in this respect becomes evident when we note that eight of 
nineteen participants in this group had thesaural difficulties. On the other 
hand 9 the six cases displaying no major difficulty in any of the three areas 
%#ere all from the tutorial group. 

Table 5.4 shows that sixteen out of the thirty-five participants 
•picked needless terms." Ihis number which was rather evenly divided between 
the tutorial and non-tutorial groups represented a majority of the cases which 
had reached the logic stage. Keeping in mind that the questions being searched 
were carefully selected to require only two terms to yield successful results ^ 
%^ find a total of 128 terms appearing at the logic point of twenty-one searches. 
Thirty of these terms were duplicates; that iSf due to mechanical errors (mostly 
mlfllnterpretation of instructions) many participants selected the same term more 
than once. On the average each participant selected six terms where two would 
have sufficed. Some participants Ignored the e3ctra or duplicate terms and 
succeeded in making correct logical combinations. Others were also able to 
complete the logic step— -though -with little apparent knowledge of the underly- 
ing concept. A few examples may be noteworthy. 
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In response to a question concerning 'Wercy death (Euthanasia) in 
New Jersey during 1973," one participant selected the term •taercy death" on 
four separate occasions. Ihus, at the point of logic, this tern was repeated 
four times with four term numbers. Ihe logiceil coniblnation made by the user 
was: •'b//l or 2 or 3 or 4" as though these were four separate files dealing 
with the same topic. This individual, as well as the one performing the next 
search to be described, was unable to select the "New Jersey" file because of 
misinterpreting "GEO" as the geography of New Jersey rather than its geographic 
location. 

Ihe second participant with the same question had selected the term 
"mercy death" twice. At the logic point seven different but incorrect logical 
combinations were made before the eighth response was accepted. The final logic 
was: "b//l and 2 and 1 and 2 and 1 and 2", terms 1 ^nd 2 both being "mercy 
death." 

On another topic, •neater pollution in the Great Lakes in 1973," one 
participant selected a total of 24 terms, seven of vrtiich were duplicates, from 
the alphabetic list of terms offered on-line. At the logic stage each one of 
these terms was linked to the others by an OR logic. Ihis individual fully 
comprehended the Boolean logic concept. She explained her search strategy on 
the basis of this assumption: as you pick terms from the on-line thesaurus 
and they aire saved for you, it seems that each term is being added on to what 
you had picked before and that in fact a logical AND connection is built up by 
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the time you reach the logic stage, l^ius the need for an OR logic at the last 
step. 

!tobles 5.6 through 5.9 reflect some of the qtiantitative data related 
to the errors made during the tutorial and the Information Bank sessions. It 
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is quite apparent that the tutorial session takes a great deal of learning 
pressures off the actual search on the Information Bank* Just as 14 out of 
16 participants from the non^tutorial group made their first error on the 
first step of the search, so did 15 out of the 19 from the tutorial group 
make that error on the first step of the tutorial. . On the other hand, this 
figure dropped to less than half the total cases when the tutorial group 
approached the Information Bank. 

On the whole, the tutorial group made fewer errors and completed 
more steps on the Information Bank than the non-tutorial group. The tutor- 
ial group averaged a minimum of 2 errors while completing an average of 8 
steps. The non- tutorial group averaged a minimum of 6 errors while complet- 
ing an average of 3 steps. 

The feeling of frustration or being "stuck" was not uncofmnon among 
the members of both groups. Six of the 19 in the tutorial group and 10 of 
the 16 in the non-tutorial group expressed jsuch feelings quite openly. Many 
commented that they would not have tried using the Information Bank without 
help, or that they would not have spent as much time trying if they were not 
participating in an experiment. Although the participants were free to end 
the session at any time, and scxne were even encouraged to do so because of 
visible signs of over-anxiety, none gave up easily once they had gotten 
Involved in the search process. 

However, two of the thirty-five participants did not wish to perform 
a search on The Niew York Times Information Bank. Both individuals had pre^ 
conceived notions about the difficulty of working with computers and felt 
that, on their first encounter, they would prefer to watch someone else work 
with the terminal. One participant who had completed the tutorial session 
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finished the first step on the Information Bank terminal ard abruptly stated 
that the *^system (is) definitely not for slow learners or old people like me.** 
The other participant simply said "You try it—I* 11 watch." 

Comments such as 'Tirst time is overwhelming," "I'm completely bewil- 
dered," and "I'm tired of reading instructions," were countered by those who 
sensed a great deal of triumph and joy in working with the Information Bank. 
One user who had spent 67 minutes at the terminal without success commented: 
"I liked itl Really enjoyed it; could stay with it all day." Another ex- 
pressed relief: "I made it throughl" Most everyone said that he vdll come 
back to use the Information Bank. But there are always those who are hard to 
please: "I would have gone straight to the printed index (of The New York 
Times) if no (human) help was provided with the system." 

Coiments on the tutorial program were generally favorable. Suggestions 
were offered for clarifying the language at the "term selection" step as well 
as "toning down" the responses given to some of the incorrect messages relayed 
by the users. Many adjustments were made in response to these suggestions. 
IWo help routines were also added at the point of term selection and logic. 
The effect of these changes, as well as a more detailed analysis of user com* 
ments on the tutorial will be studied in chapter 7. 
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Table 5.1 



Background Information on Participants In the Feasibility Study 

Number In Number In 

Topic Tutorial Group Non«»tutorlal Group Total 

Sex; 

Female 17 14 31 

Male 2 2 4 

Last degree obtained: 

Bachelor's 16 14 30 

Other 3 2 5 

Educational background: 

Social sciences/humanities 18 15 33 

Science 1 12 

Previous computer experience: 

Any contact (batch or on-line) 3 5 8 

On-line use (time-sharing) 1 3 4 

Familiarity with The New York 

> Times Printed Index 9 5 14 

Number of participants 19 16 ' 35 



Table 5.2 

Total Time Spent in Completing the Tutorial Program 
Number of Minutes Number of Cases 



16-20 


2 


21-25 


5 


26-30 


5 


31-35 


2 


36-40 


1 


41-45 


3 


46 and over 


1 
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Table 5«3 

!Zbtal Time Spent Searching The Ifew 

Number of Cases 
Number of Minutes (tutorial group) 



u 


0 


1 c 


1 




0% 

2 




5 


1 r OA 

Xo^^JU 


3 


21-25 


2 


26-30 


1 


31-35 


2 


36-40 


3 


41-45 


0 


46 and over 


0 


Total 


19 
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Times Information Bank 



Number of Cases 

(non-tutorial group) Ttotal 

2 2 

1 2 

0 2 

0 5 

4 7 

0 2 

1 2 

2 4 
2 5 
2 2 
2 2 

16 35 



T^le 5.4 

Search Results 

Number of Cases NUmber of Cases 
Observation (tutorial group) (non-tutorial group) itotal 

Search **mechanically" ^ 
completed with by-pass* • 2 17 

Search 'Mechanically*' 
complete* No reservations** 12 3 15 

Accepted negative result as 

final output 3 1 4 

Picked needless terms 9 7 16 

Search successful 9 0 9 

* This group Includes those cases where pne of the steps such as 
modification or logic may h^ve been bypassed 

This group includes only tJhose cases which completed every step 
necessary for a successful search 

Table 5.5 

Dominant Factor in Search Falliires on 
The New York Times Information BaiJc 

Number of Cases NUmber of Cases 
Type of Probleici (tento rial group (nor^t- tutorial group ) Total 

Conceptual 4 10 14 

Thesaural 8 3 II 

Instructional - 2 2 

No dominant problem 

(successful .'Search ) 6 «. 5 

Participant did not wish ^zo 
conduct a search 1 12 

Total 19 16 35 
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Table 5*6 



NY!nB iScreffl 

1] Identification 

2: S/steni Nesflagea 

3^ Proceed 

4.' Term Entry* 

IS 5: Terra Not in Pile 

6i Term Selection 

7i XRP, etc. 

8: Hodiflcation 

9j Logic ' 

Made No Significant 
Mechanical Error 

Strictly Thesaural Error 

Did Not Participate in 
NYTIB Search 



First Significant Error Made in the Courae of the Session, 
Including Those Which Hay Have Subsequently Been Corrected 

# of Cases in the # of Cases on NYTIB # of Cases on NYTIB total 



Tutorial Session By the Tutorial Group By the Non^tu' 



15 
n/a 

4 
n/a 



2 
1 

3 
2 



n/a 



:orial group on NYTIB 



22 



• Term entry is the first NYTB screen shown in the tutorial session 
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Table 5.7 

last Step Successfully Completed on !Ihe New York Times Information Bank 

Number of Cases Number of Cases 
Screen (tutorial group) (non~tutorlal group) Total 

Did Not Complete Any 

Step Successfully - 2 2 

1: Identification 1 3 4 

2: System Messages «- - - 

3: Proceed 1 12 

4: Term Entry - - - 

5: Term Not in File - - - 

6: Terra Selection 2 3 5 

7: XRP, etc. - - - 

8: Modification 1 - 1 

9: Logic 4 2 6 

10: Viewing Abstracts 10 5 15 

Total 19 16 35 

Table 5.8 

Total Number of Significant Errors Made During the Tutorial Session 
Number of Errors Number of Participants 

2 1 

3 , 1- 

4 1 

5 7 

6 2 

7 2 

8 1 ' 

9 1 
10 and over 3 
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Table 5.9 

Total Number of Significant Errors Made During One Session on The NYTIB 



Number of Cases Number of Cases 

Number of Errors (tutorial group) (non-tutorial group) TPtal 

0 5 - 5 

15 16 

2 3 1 4 

3 2 3 5 

4 1 12 

6 1 - 1 

7 1 2 3 

8 - 2 2 

11 and Over 1 5 6 

Participant did not wish to 

conduct a search on NYTIB - 11 
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CHAPTER VI 

IHE FORMAL EXPERIMENT! ANALYSIS OF DATA 

The feasibility of conducting an experiment to test the effectiveness 
of the CAI program as a teaching tool for The New York Times Information Bank 
was confirmed in a preliminary study using thirty«>flve subjects from the 
Graduate School of Library and Information Sciences* In order to establish 
the grounds for generallzablllty of the findings , it was decided to open the 
e9q>eriment to the academic community at large, and request that each particle 
pant bring with him a question of genuine interest or use to himself. Also, 
to further differentiate between the thesaural problems encountered by a 
novice user and the mechanics of conducting a search, the formal experiment 
was divided into two sections. Design of the experiment for Section I closely 
followed that of the feasibility study, except that each participant was assis-* 
ted in the off-line term selection process. Section II was given no human 
assistance beyond the general introduction to the keyboard and equipment and 
the logging in and out of the tutorial program. 

A total of 75 appointments were made, eleven of which were sxobsequently 
cancelled or eliminated from the final analysis. The first forty cases com- 
prised Section I and the remaining cases formed Section II of the formal exper- 
iment. Because of the conduct of the experiment in two sections and the varia- 
tion in treatment of the subjects in Section I and II, the results will be 
analyzed separately. On the other hand it must be noted that such variations 
did not affect the underlying concept or the basic structure of the design. 
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Ihe* sijnllarity in results among the members of the feasibility study, Sections 
I and II was confirmed by a Chl-square distribution equalling 1#673 at a .05 
level of significance for two degree^ of freedom. Furthermore, a contingency 
coefficient of the Chi-square (Cy^—e-) showing a .17 variation among the three 
sections also verifies the internal stability in the results obtained from each 
section* 

The formal experiment was designed on the basis of the findings of the 
feasibility study. Individtoal groups were to be kept small enough to allow for 
personalized and extended record-keeping on the behavioral as well as physical 
reactions of each participant before, during, and after a search. Such records , 
by their nature, tend to contain many subjective factors. Ihe intent of the 
following analysis is to explore such factors and use the data gathered only as 
a tool to delve into the qualitative characteristics of a first*-time search on 
The New York Tiroes Information Bank. 
Section I 

Ihe forty participants in this part of the experiment were evenly divided 
between the tutorial and non-tutorial groups. Table 6.1 shows the obvious simi- 
larity in the background and makeup of the two groups* The only slight varia- 
tiai seems to be In the participants' previous experience with the computer. 
As with the feasibility study, the random assignment of individuals placed a 
larger number of those with a computer background in the non-tutorial group. 
However, this difference of four out of forty proved insignificant In terms 
of the final search results. 

Ihe time spent by the non-tutorial group in reading printed instruc- 
tions was obtained for both sections of the formal experiment. Table 6.2 
shows that the majority of the non-tutorial group spent less than sixteen 
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minutes in reading the printed Instructions , and that no one spent over thirty- 
five minutes on this task. The tutorial groupi on the other handf averaged 
about thirty-two ndnutes In completing the CAI session* Observing the combined 
figures for both groups reveals that thirty-three out of forty participants 
felt ready to start an on-line search with less than thlrty-slx minutes of 
preparation time. 

Regardless of the Instructional medium used during the training period, 
no participant spent more than thirty minutes on Ihe New York Times Information 
Bank, with an average of about ten minutes per search (see table 6.3 )• 

Although the typical participant from the non-tutorial group was refers 
ring to the printed User Guide while condxjcting the on-line search, it is Inter- 
estJLng to note that on the average he spent three minutes less than his counter^ 
part from the tutorial group working with the Information Bank. Of course, 
table 6. 7 reveals that over half of the non-tutorial group did not go beyond 
the term selection stage, which may explain the shorter time spent at the termi- . 
nal. Table 6.4 also suggests that the time spent at the terminal by the non- 
tutorial group was not as fruitful as that of the tutorial group* While eleven 
participants (or over half) of the tutorial group succeeded in retrieving the 
desired results from the Information Bank, only three out of twenty individuals 
relying solely upon written instructions were eJble to achieve the same results* 
Participants in both groups displayed symptoms of conceptual confusion 
as ^rell as instructional misinterpretation* Although all p^urticipants in 
Section I of the experiment had been guided in their term selection processes, 
four individuals still showed dominant thesaural problems* One simply claimed 
that thesaurus terms were "ambiguous," another mistook the "geo" modification 
for the geography of the nation he was interested in, and a third picked a 
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duplicate file (Indexing error) of an acceptable Index term with only a hand- 
ful of citations* Thm fourth case involved a shift from the participant's 
original question on "photography" to one dealing specifically with "Kir Han 
photography 9 " Much of this Individual's time was spent In searching the 
thesaurus for a file dealing specifically with this topic • 

The conceptual problems faced by the participants varied widely, with 
most Instances identified in chapter 5 being represented • Again we find that 
misinterpretation of instructional messages was present In varying degrees* 
IWenty-seven of tKe forty cases displayed various symptoms related to this 
problem, with eighteen Individuals expressing hardship in Interpreting the 
phrase '*vlew continuation" or distinguishing between the A and B options In 
term selection and logic stages* 

In comparing the data from tables 6*6 and 6«9 with similar figures 
from the feasibility study (tables 5«6 and 5«9}, we find an obvious change 
in the reduced:, number of errors on the Information Bank. Three reasons may 
be cited to explain this phenomenon* First is the change in the setup of the 
formal experiment* The unusually high incidence of errors at the identifica- 
tion stage occuring during the feasibility study led to a permanent addition 
of a sign to the CRT unit* Placed directly above the top left comer of the 
screen, this sign Informs all potential users of the system that they must 
use the University of Pittsburgh identification number, and explains how 
this number is to be typed and reminds the user to press the "enter" key* 
While in the feasibility group twenty-two of the thirty-five participants 
erred at this stage, this numbtir was drastically cut down to only eight out 
of forty cases for Section I of the formal ejq^erlment* 
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A change in the design of the experiment was also responsible for 
the lower rate of errors. Section I had the benefit of discussing the 
selection of appropriate terms for each inquiry before the on-line session 
at the Information Bank was begun. Ihis procedure eliminated much of the 
trial and error process by which the participants in the feasibility study 
managed to select terJs from the on-line thesaurus. 

A' more significant reason for the reduced number of errors was the 
general attitude of the experimental group. Acting more like the general 
user population of Ihe New York Times Information Bank than an "experimental" 
group, the participants displayed all the typical characteristics of the 
Individual with a real information need. They were interested in the end- 
result of the search and not curious about the ••machine" beyond the level of 
necessity required by the task. Ihis meant that their tolerance level for 
receiving error messages and confusing instructions was relatively closer 
to the normal laser population, resulting in their "giving up" without too 
many false attempts at interpreting system messages. Participants in 
Section I made an average of 1.7 errors per search and the highest number 
of errors in any search was five. On the other hand the participants in the 
feasibility study averaged, a minimum of 4.2 errors per search with six cases 
making over eleven errors, (see table 5.9)« 

As may be expected, there was little variation between the performance 
of the participants in Section I and the feasibility group on the CAI program. 
Each shewed an average of about six errors per session. 

On the whole, participants in Section I were the most successful in 
obtaining desired results. Fourteen out of forty successfully completed a 
search on the Information Bank. Of this fourteen, eleven had participated in 
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the CAI training session while three had used only printed instructions.. 
OSiia figure compares well with the success rate of nine out of thirty-five 
in the feasibility study, and five out of twenty-four for Section II of the 
experijnent« 

It may thus be concluded that the CAI session is roost fruitful for 
teaching conceptual, mechanical and instructional characteristics of the 
Information Bank when it is combined with a live discussion on the term 
selection process. Findings related to Section II introduce a new perspec- 
tive to this conclusion. 

Section II 

This was the smallest section of this experiment, consisting of 
twelve individuals in each of the tutorial and non-tutorial groups. Because 
of its particularly small sample size, it would be improper to draw general- 
izable conclusions from this data. However, upon further reflection, one 
cannot overlook the fact that the -success rate for this section has shown a 
decline from those reported for Section I and the feasibility study. Tible 
6.20 provides the basis for such comparison. As noted earlier, the contin- 
gency coefficient of the Chi-square distribution has shown this difference to 
be statistically insignificant. That is, this decline may be attributed to 
chance alone. But in order to account for all differences in the conduct of 
the experiment we may wish to further explore the fact that individuals in 
this group received no assif;Vai^e during their off-line term selection pro- 
cess* One may intuitively cc^ .Vade from this observation that an overvrtielming 
number of participants in Section II faced thesaural problems. Closer examina- 
tion, however, reveals that only four of the twenty-four searches were aborted 
due to thesaural problems (see table 6.14). Excluding the five successful 
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searches, we find fifteen participants who "gave up" becaiise of conceptual 
or Instructional rather than thesaural problems. 

An examination of the conceptual problems shows that fotur out of ten 
individuals were looking for a logical combination of their terms at the term 
selection stage. While-, the participants may have entered acceptable index 
terms into the system, thus displaying no evident thesaural problems, they 
were in fact unable to conceptualize the flow of steps involved in the search 
which could enable them to select their terms frorr* the on-line display. How- 
ever, participants in Section I also displayed the same level of misconception, 
where four of the eleven cases with conceptual problems faced the same dilemma. 

Having established the comparability of the two sections In all res- 
pects but for the off-line term negotiation stage, we may conclude that there 
\iere other "hidden" values in such a stage, lhat is, the sheer human contact 
and interaction, and establishing rapport between the participant and the exper- 
imenter (who is in this case also the "authority") may have played a more signi- 
ficant role in conducting the search than merely providing assistance in term 
selection. 

Participants in Section II came closest to doing a search in the 
absence of an attendant, yet the only spontaneous comments presented upon com- 
pletion of a session referred to the quality of assistance offered by the 
attendant. Of course, this assistance was offers after the participant had 
completed an un-aided search and refers to the full explanations given during 
the parallel search. The significemce of such comments is that it re-enforces 
the author's belief that individual users naturally identify with the •'human" 
element of the system. It may be worthwhile to repeat this portion of the 
experiment (Section II) on a larger scale to establish the significance of 
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"human contact" as eui essential criterion for conducting a successful first- 
time search on The New York Times Information Bank« 

Characteristics of the Successful Search 

A total of ninety-nine searches were conducted under experimental con- 
ditions between November, 1973, and April, 1974, in three separate groups. 
Although the experiments were designed to test certain specific aspects of 
conducting first-time searches on The New York Times Information Bank, the 
ultimate goed was the same in all cases: that of training the novice user to 
conduct a successful search. For the purpose of these experiments a "success- 
ful search" was defined as one which satisfied the user*s immediate information 
needs. Except for the feasibility study which required the participant to find 
citations for a question chosen by the experimenter, the participant's Judge- 
ment on the usefulness of the end result was taken as final. Because the ques- 
tions posed during the formal experiment were supplied by the participant and 
were meant to satisfy his personal needs, there were a small number which did 
not match the '•normal search" profile as defined in chapter 3. Although all 
questions were screened before an appointment was made for the experiment, 
many questions were subsequently revised as the participant began to conduct 
the actual on-line search. The small size of certain files prohibited the use 
of modification in some searches. Thus, even though some searches may not have 
•Mechanically" fulfilled the requirements of the experiment, they were nonethe- 
less successful in answering the user's question. 

Of the ninety-nine searches completed during the course of these expert 
iraents, twenty-four were mechanically complete as well as yielding successful 
results. Twenty- three of these twenty- four searches were conducted by the pax>- 
ticipants in various tutorial groups*. Four other cases where the modification 
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stage was bypassed but the search was successf' 1 were added to the previous 
twenty-four cases to make a complete record of all sxKZcessful searches. Four 
of these twenty-eight cases are from non-tutorial groups. Table 6.19 shows 
the characteristics of a successful search and the success rate for each set 
of these characteristics* The success rate Is defined as the ratio of success- 
ful cases beaurlng a particular characteristic to the total number in the exper- 
Ijnental group sharing that characteristic. 

In order to obtain the probability of success for the sample population 
at the .95 confidence level, standard error for the sample was calculated. Ihe 
results show the probability of success (p) for the true population to be: 

.20 < p <.37 

niat is, all rates of success falling within the boundaries of .20 and .37 are 
statistically insignificant and the differences may be attributed to chance, 
table 6.19 shows all but two of the user characteristics falling within these 
boundaries. 

The only significant figure is the nearly six-fold success rate of the 
tutorial over the non-tutorial group. From a different perspective, vre find 
that %rtiile the entire group had a 28% success rate (28 cases out of 99), those 
in the tutorial group displayed a 47% success rate. Wiis confirms the study^s 
hypothesis that a combination of the CAI program and printed instructions is a 
more effective training tool than the printed instructions used alone. 

All other figures in table 6.19 fall within iP*^ of the group's success 
rate, showing little significance in the effect of other observed user charac- 
teristics on search results. 

On the average, we find that the successful searcher spent between 
16-*30 minutes on the CAI session, and that his search on the Information Bank 
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took about 14 rtdnutes. He has a 50% chance of mi sunder standi Jig some of the 
system messages* It is quite likely that these will be related to the 'View 
continuation" phrase or the A/B options at the term selection and lo^ic stages. 

Hie successful first-time searcher makes an average of 1.4 significant 
errors per search. These errors may occur any time during the negotiation pro-* 
cess^ but he is able to correct them vdthout any assistance from another indi- 
vidual. However^ there is no indication that the successful seeurcher would 
have approached the terminal on his own and conducted a search in the absence 
of an attendant. Many users expressed their apprehension of dealing with a 
fully mechanized system without any human assistance within easy reach. 
Results of Section II of the experiment, where the participants had the least 
amount of human contact, seem to confirm this experience. 
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Table 6.1 



Backgrou.id Characteristics of Participants in Section I 



Topic ' 

Sex: 
Male 
Female 

Last Degree Obtained: 
High School Diploma 
Bachelor's 
Master's 
Ph.D. 

Educational Background: 
Social Science/Humanities 
Science 

Previous Computer Experience: 
Any Contact (batch or on-line) 
On-line (time-sharing) 

Familiarity with The New York Times 
Printed Index 

Number of Participants 



Number ixi 
Tutorial Group 



14 
6 



8 
6 

5 
1 



17 
3 



9 

8 



8 

20 



Number in 
Non-tutorial Group 



15 
5 



7 

5 
8 



17 
3 



13 
12 



8 

20 



Total 



29 
11 



15 
11 
13 
1 



34 
6 



22 
20 



16 
40 
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Table 6.2 

Section I: Total Time Spent in Preparing for an On-line Search (training period) 



Number of Minutes 

Less than 16 minutes 

16-20 

21-25 

26-30 

31-35 

36-40 

41-45 

46 and Over 



Number in 
Tutorial Group 



2 
4 

6 
1 
2 
3 

I O 2 



Number in 
Non-tutorial Group 

13 

2 

1 

1 

3 



63 



l^le 6.3 

Section I: Total Time Spent Searching The New York Tiroes Infonmation Bank 

•Number of Cases Number of Cases 



Number of Minutes 


(tutorial qroup) 


(non- tutorial 


group ) 


Total 


0 


1 


1 




2 


1-5 


- 


5 




5 


6-10 


11 


8 




19 


11-15 


3 


4 




7 


16-20 


3 


1 




4 


21-25 


1 


1 




2 


26-30 


1 






1 


Total 

• 


20 

Table 6.4 


20 




40 


Section 


I: Search Results 








Observation 


Number of Cases 
(tutorial group) 


Number of Cases 
(non-tutorial croup) 


Total 


"Mechanics" of Search 
Completed In Full 


12 


2 




14 


"Mechanics" of Search 
Completed With By-pass* 


2 


4 




6 


Search Fully Successful 


11 


3 




14 


Search Partially Successful 


1 


1 




2 


Any Difficulty With On-line 
Instructions • Interpretation 


12 


15 




27 



• By-passed modification or logic stage which was not essential for the 
question at hand 
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Table 6.5 



Section I: Dominant Factor In Search Fedlxires 



Type of Problem 

Conceptual 

Thesaural 

Instrxictlonal Interpretation 

No Dominant Problem 

Participant Did Not Wish to 
Conduct a Search 

Total 



Number of Cases Number of Cases 
(tutorial group) (non-tutorial group) Ototal 



3 
1 

3 

12 

1 

20 



8 

3 
5 
3 

1 

20 



11 
4 
8 

15 

2 
40 
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Table 6.6 

Section I: First Significant Error Iteide in the Course of a Session, 

Including Those Which May Have Subsequently Been Corrected 



NYTIB Screen 

1: Identification 

2: System Messages 

3: Proceed 

4: Term Entry 

5: Term Not in File 

6: Term Selection 

7: Cross Reference, etc. 

8: Modification 

9: Logic 

Made No Significant Errors 
Did Not Participate in Search 
Total 



Number of Cases Number of Cases 
(tutorial group) (non-tutorial group) itotal 
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7 
1 
3 
1 
4 
1 
20 



7 
3 



1 

3 
1 

20 



8 
3 



9 
1 
3 
2 
7 
2 
40 



Table 6.7 

Section I: Last Step Successfully Completed on The New York Times 
Information Bank 

Number of Cases Number of Cases 
NYTIB Screen (tutorial group) (non-tutorial group) Total 



No Step Successfully Completed 1 2 3 

1: Identification .4 4 

2: System Messages - 11 

3: Proceed - 1 1 

4: Term Entry 2 13 

5: Terra Not in File - 11 

6: Term Selection - 11 

7: Cross Reference, etc. - 11 

8: Modification - 1 1 

9: Logic 3 14 

10: Viewing Abstracts 12 5 17 

Total 20 20 40 
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Table 6.8 

Section I: Total Number of Significant Errors Made During the Tutorial Session 
Number of Errors Number of Participants 

1 1 

2 2 

3 1 

4 2 

5 1 

6 6 

7 1 
9 1 

10 and Over 4 

Data Not Available 1 

Total 20 

Table 6.9 

Section I: Total Number of Significant Errors Made During One Session on 
The New York Times Information Bank 

Number of Cases Number of Cases 
Number of Errors (tutorial group) (non-tutorial group) Total 



No Significant Errors Made 


4 


4 


8 


1 


8 


3 


11 


2 


2 


6 


8 


3 


1 


4 


5 


4 


4 


1 


5 


5 




1 


1 


Did Not Participate in Search 


1 


1 


2 


Total 


20 


20 


40 
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Table 6.10 

Background Characteristics of Participants in Section II 



Number in Number in 

Topic Tutorial Group Non-tutorial Group Total 

Sex: 

Male 9 6 15 

Female 3 6 9 

Last Degree Obtained: 

High School Diploma 5 5 10 

Bachelor's .6 . 3 9 

Master's 1 4 5 

Educational Background: 

Social Science/Humanities 10 11 21 

Science 2 13 



Previous Computer Experience: 

Any Contact (batch or on-line) 5 3 8 

On-line (time-sharing) 4 2 6 

Familiarity with The New York Times 

Printed Index 9 9 18 

Number of Participants 12 12 24 

liable 6.11 

Section II: Total Time Spent in Preparing for an On-line Search (training period) 

Number in Number in 

Number of Minutes Tutorial Group Non-tutorial Group 

Less than 16 minutes - 4 

16-20 1 1 

21-25 2 3 

26-30 3 3 

31-35 

36-40 1 1 

41-45 1 

46 and Over 4 - 
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Table 6.12 

Section II: Total Time Spent Searching The New York Times Information BanJc 



Number of Minutes 

1-5 

6-10 
11-15 
16-20 
31-35 

Total 



Number of Cases Number of Cases 
(tutorial group) (non-tutorial group) Total 



3 
3 
2 
3 
1 
12 



6 
3 
3 



12 



9 
6 
5 
3 
1 
24 



Table 6.13 
Section II: Search Results 



Observation 

••Mechanics" of Search 
Completed in Full 

"Mechanics" of Search 
Completed With By-pass* 

Search Fully Successful 

Thesavunis Problems 

Any Difficulty With On-line 
Instructions 



Number of Cases Number of Cases 
(tutorial group) (non- tutorial group) Total 



2 
4 

2 



2 
1 
3 



4 
5 
5 

13 



• By-passed modification or logic stage which was not essential to the search 
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Table 6.14 

Section II: Dominant Problems in Conducting a Search on the Information Bank 



Type of Problem 

Conceptual 

Ihesaural 

Instructional Interpretation 

No Dominant Problem 

Participant Did Not Attempt 
Independent Search 

Cause of Search Failures: 
Ihesaural 
Non-thesaural 



Number of Cases Number of Cases 
(tutorial group) (non-tutorial group) Total 



4 

3 
3 
2 



2 
6 



6 
2 
2 
1 



2 
9 



10 
5 
5 
3 



4 

15 



Table 6.15 

Section II: First Significant Error Made in the Course of a Session, 

Including Those Which tlay Have Subsequently Been Corrected 



NYTIB Screen 

1: Identification 

2: System Messages 

3: Proceed 

4: Term Entry 

5: Term Not in File 

6: Term Selection 

7: Cross Reference, etc. 

8: Modification 

9: Logic 

Made No Significant Errors 
Total 



Number of Cases Number of Cases 
(tutorial group) (non~ tutorial group) Total 



2 
1 
2 



1 
2 
12 



8 
1 



a 

3 

1 

2 



12 



2 
2 
24 
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Table 6.16 

Section II: Last Step Successfully Completed on The New York Times 
Information Bank 



NYTIB Screen 

No Step Successfully Completed 
1: Identification 
2: System Messages 
3: Proceed 
4: Term Entry 
5: Term Not in File 
6: Term Selection 
7: Cross Reference, etc. 
8: Modification 
9: Logic 
10: Viewing Abstracts 
Total 



Number of Cases Number of Cases 
(tutorial group) (non- tutorial group) Total 



1 
1 

3 
1 



5 
12 



3 
2 



2 
1 



1 

2 
1 
12 



3 
3 
1 

5 
2 



2 
2 
C 
24 
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Table 6.17 

Section II: Total Number of Significant Errors Made During the "Aitorial Session 
Nupber of Errors Number of Participants 

1 



4 
5 
6 
7 
8 
9 

10 and Over 
Data Not Available 
Total 



3 
2 
1 
2 
1 
1 
1 
12 



Table 6.18 

Section II: Total Nuniber of Significant Errors Made During One Session on 
The New York Times Information Bank 



Number of Errors 

No Significant Errors Made 

2 
3 
5 
6 

Total 



Number of Cases Number of Cases 
(tutorial group) (non-tutorial group) Total 



2 
6 
1 
1 
1 
1 
12 



4 

6 

2 



12 



2 
10 
7 
3 
1 
1 
24 



8 i 
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Table 6.19 

Success Rate for Individual Characteristics 



Characteris tics 



Male 
Female 

Undergraduate 
Graduate 

Any Form of Computer 

Experience 
SF>ecifically On-line 

Experience 

Familiarity With the Printed 
Index 

Used CAI Training 

Used Printed Instructions 



(A) (B) 
Group of Total 

Successful Searchers Expeirime n tal Group 
T=28 T=2";) 



16 
12 

7 
21 



13 

11 

17 
24 
4 



48 
51 

26 
73 



38 

30 

48 
51 
48 



(C) 
Success 
Rate 
A/B 

.33 
.23 

.27 
.29 



.34 
.36 

.35 
.47 
.08 



Table 6.20 

Comparison of the Success Rates in Each of the Ihree Experimental Groups 



Elxperiment 
Feasibility Study 
Section I 
Section II 



Number of 
Successful , 
Cases 

9 

14 

5 



Total number 
of Cases 

35 

40 

24 



Success Rate 
Tutorial rfon-tutorial 



0.47 
0.55 
0.33 



0.0 

0.15 

0.08 
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CHAPTER VII 
A COMMENTARY ON THE INSTRUCTIONAL MEDIA 



The formal experiment was originally designed to test the effective- 
ness of the computer-assisted instruction in conjunction with the printed 
instructions against the printed instructions alone. But most participants 
preferred to stay with only one form of instruction throughout the experi- 
ment. Those who had taken the tutorial used the printout of that interaction 
instead of referring to the User Guide for assistance. In effect the experi- 
ment became a test of CAI versus the User Guide . The only common medium for 
the two groups was the "blue card" — a 4" by 9" card summarizing the stages of 
each search and explaining the universal options available on-line. This card 
which was distributed to all participants as an introduction to the system •s 
operation was designed to complement the CAI program and appears in Appendix B. 

The "blue card" performed a very small function in the actual training 
process and elicited few comments* There is no conclusive evidence that it 
either helped or hindered a participant's ability to conduct a search on The 
New York Times Information Bank. Because of the inconsequentiality of its 
role in training users, this piece of printed instruction will not receive 
further attention in this chapter. Following the natural division of the two 
groups, the remainder of the chap^-er will deal with the CAI program and the 
User Guide in separate sections. 
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I* The CAI Program for itie New ^York Times Information Bank 

During the feasibility study all interactions with the tutorial pro- 
gram were reviewed for obviou ■ programming errors as well as key questions 
which elicited an unusual number of wrong responses. A typical interaction 
printout is included in Appendix B. The two major stages where users made 
their first incorrect response were earmarked for programming changes. 

Although many individuals made their first error at the term entry 
stage (following the NYTIB formatting rules), most were able to correct that 
error on a second chance offered by the program. The next step— that of term 
selection from a thesaural display — was the most confusing step '^or all users. 

Difficulties in term selection from the menu offered on the "screen" 
ranged from simple negligence in reading all options and explanations offered, 
to serious misinterpretation of instructions. It was obvious, however, that 
t}\e explanations offered were not sufficient for most individuals. Thus, a 
•Tielp" routiJie was added to this section, allowing users to get better 
acquainted with some of the system terminology. 

Help routines for the term selection and logic stages were availabl 
to all [.^ticipcints in the formal experiment. Twelve out of twenty individ- 
uals in S€x:t^.cr: jl, and five out of twelve in Section II referred to these 
options. «?iire thost inc^ividuals, however, who did not ask for help 

even thoug' tney f eJ ^ thi\t the instructions were unclear and that they 
needed assistance in def irunr? .tiessages such as "view continuation" or 
'^citation range*" One such iixiividuel offered his reason for not asking for 
h'i'lp: "I don't wemt to admit to the computer how dumb I am." Of course, 
this may be interpreted as reluctance on the part of the participant to admit 
his lack of understanding to the experironter. Two participants insisted upon 
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keeping: nrlTitout of their Interaction on the tutorial program. Othercj 
felt obl'^ ced to justify their incorrect responses by indicating that it was 
done c . cpose to find out how the computer would react to unexpected respon- 
ses. One participant was totally unable to grasp the concept of the program 
or the "flow" of steps involved, repeating the first correct response suggested 
by the program throughout the remainder of the session. 

Many of the conceptual problems faced by users of the Information BanJc, 
and discussed in chapter 5, were encountered during the tutorial session as well. 
Some felt that a review of the entire printout vnon completion of the session was 
the best way for understanding the flow of steps and their interrelationships. 

It is quite difficult to judge the effect of the CAI program on individ- 
ual participants. Given a single individual, there is no feasible way to test 
his understanding of the Information Bank's system messages before and after the 
tutorial session. Once he has been exposed to the system, the residual effects 
will carry over and affect his performance on the tutorial as well as his sub- 
sequent performance on the Information Eiank# 3y th^ same token, if he were to 
complete the CAI session before his first exposure to the Information Bemk—as 
was the case in this experiment — it cannot be concluded that the tutorial ses- 
sion was the ONLY reason for his success or failure in conducting an actual 
search. 

TaKen as group, however, it becomes obvious that those exposed to 
the CAI training session stand a better chance in successfully completing an 
unassisted search on the Information Bank. Results of the formal experiment 
show clearly that this sticcess is not due to the participant's exposure to an 
on-line system alone. Those with previous experience with computers and on- 
line systems unrelated to The New York Times Information Bemk showed no 
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significant advantage over those with no computer background. Ihus, the sig- 
nificance of the tutorial progrv=un created for this particular system camnot 
be denied. However, there were many shortcomings inherent in the design of 
this training session. 

The major deficiency of the program was its incapability to build up 
error histories for individual users. For example, in the case of the indi- 
vidual whose single response was unvaried throughout the entire session, the 
program was \inable to detect this error and offer individualized assistance. 
Also, the program's error 2inalysis worked within a limited r.ange of user res- 
ponse and did not provide for all conceivable errors which may have been made 
at a given point. 

Criticisms of the tutorial were offered on three levels: technical, 
structural, and intellectual. From a practical viewpoint suggestions were 
offered in c:pand?,ar \he scope of definitions and explanations of the system 
messages at vJiU'loa*:. stagf*5^ Many of these, suggestions have been incorporated 
in the upjJjit'S^i iVii'^Jon oi the program. The revised program also reflects the 
rhanges which have taken place on the Information Bank system messages in 
'j^ViC-^rit months, ^ese changes include the recognition of certain common tech- 
nical errors made by users such as placir..; an unnecessary double slash (//) 
formatting symbol at the end of a command. 

Being a simulation of the Information Bank, the tutorial was subjected 
to the same structural criticism directed toward the actual system. It has 
been stated that there is ambiguity in the flow of steps and that not enough 
explanation is offered on each "screen". It has been suggested that a synop- 
sis of the contents of the program should appeair at the beginning (as well as 
the end) of the session. Although such criticism is quite justified, it must 
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be noted that such additions would add to the total session time without 
Increasing the user/system interaction. Because of this consideration, the 
summary is presented as a supplementary printed card (the ^'blue card'*) which . 
is available to all users of the system. 

Exparsion of explanatory messages would also lessen the active role 
of the user in the program. In order to add to the interaction, the total 
time of each session would have to be expanded. Experience shows that the 
first- time user of the Information Bank cannot "comfortably" devote over 
thirty minutes of his tiine to the preparation/learning process before a 
search. 

Most of the above suggestions were taken into consideration, however, 
in developing the second CAI program which deals strictly with the process of 
term selection and entry. 

On an intellectual level, the major objection to the tutorial program 
was its very restricted se.irch capabilities. The session was specifically 
designrjd to answer a "typical" question—comprised of two terms, a date modi- 
fiei:,» aivJ an AND logical connective. One user objected that he was being 
forcr to view t2ie question from the programmer's point of view; he later 
generalized his confluent to all CAI activities. Many others <;howed through 
their choice of responses that they would have preferred to deviate fra^i the 
question at hand ar.d explore other options offered on the "screens". For 
example, many insisted upon selecting more than the two required terms, or 
the date modifier. 

Hiwartir j the user's curiosity to explore other p>ossibilities through 
this training session was noted as the most frequent o/?jec: . - raised by the 
participants in the experiment. But tire limitations, both the author's oiid 
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the user*Sf was a cr\x:ial factor in determining the length and scope of a 
first-time orientation program for the Information Bank. 

In analyzing the success rate of the tutorial group in the feasibil- 
ity study, it was found that fourteen out of seventeen individuals who were 
able to complete the term selection stage on the Information Bank went on to 
complete the '^mechanics" of a search. looking at the group as a whole, this 
statement holds true for 21 or 27 cases completing the term selection process. 
Ihis as well as other evidence points toward term selection as being a major 
hxirdle in the search negotiation process. Steps were taken to isolate common 
mistakes by first- time users leading to failure in term selection. A 

second CAI program has been written to deal with these mistakes and to give 
hints on selecting terms off-line, procedures for entering these terms into 
the system, and finally selecting the correct files from the on-line thesaurus* 
A typical sample interaction Is included in Appendix B. Although this ••drill- 
and-practice" program has not been tested in a controlled environment, the user 
response so fair indicates that it is a useful addition to the training process. 

Ihe CAI program sets a slower pace than the Information Bank itself. 
It corrects certain errors and offers explanatory messages for various steps 
of the search. It helps the novice user get acquainted with an on-line system 
on a one-to-one basis, yet poses less of a threat than the Information Bank. 
This is especially true for cost-conscious individuals who become over-anxious 
about the comm\inication costs between Pittsburgh and New York, However, des- 
pite the advantages of the CAI program and its obvious benefits for at least 
half of the user group, it is the author's strong feeling that all users can- 
not be, and should not be subjected to the same training program. Some indi- 
viduals may benefit more from traditional training methods. Familiarity with 
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the instructional medium establishes the common grounds necessary to start the 
leaumlng process and removes some of the basic threats which a computer program 
may pose to the uninitiated user. 

11. Printed Instructions 

Three forms of printed instructions for the Information Bank were 
available at the time of the experiment. Additions and changes have been made 
since March of 1974, and the author has been informed that a new guide is under 
preparation. However, as of October, 1974, these three basic tools are still 
being used at the Information Bank. One is a comprehensive loose-leaf guide 
of over 300 pages which, though suitable as a reference tool, is rather cumber- 
some to utilize by the casual user of the systeni. There is a shorter (32 page) 
User Guide which was used as the main medium of instruction for the control 
group of the experiment. An even shorter guide (12 pages) entitled User Guide; 
helpful hints about the incyuiry process , which is a companion to the User Guide , 
was also placed at the disposal of the F>articipants in the feasibility group. 
It was subsequently removed from the formal experL-^^t because of the poor 
reception it received during the preliminary study. It seems that this abbre- 
viated guide is too short to provide all necessary information for a first-time 
search, yet too long and monotonous to capture tne attention of those who have 
already scanned through the other User Guide . 

Ihus, the commentary on printed instructions accompanying Ihe New York 
Times Information Bank is limitea to the 32-page version of the User Guide . 
This guide is reproduced in full as Appendix A of this study. 

The advantages of using some form of printed instruction in ej\ on-line 
system were discussed in chapter 2. As a summary of that discussion, the most 
favorable points for the use of this medium m^^y be I'^isted as the familiarity 



of the user with this fonn of instruction; and its permanence, portability, 
and independence from external conditions. That is, the user is able to carry 
a manual with him to study il at his leisure. He is able to refer to it over 
and over again. His preliminary use of the guid€^~to learn about the system~ 
is not dependent upon the availability of the system itself. It will also cost 
less to obtain and read a manual than to operate a CAI program. 

Other advantages of a user guide for an on-line system include the cap- 
ability of using the instructions in conjunction with the system as opposed to 
the sequential use of a separate CAI program. A printed guide can go into more 
detail than CAI; it is also more suitable as a medium of instruction for those 
who feel uncomfortable with computers. 

Yet we find a number of shortcomings in the use of printed instruc- 
tions as well. Generally speaking, j>eople do not read instructions very care- 
fully; they do not take the time to read every detail. Transfer of information 
from printed form to action is rather difficult for many individuals. We also 
face the added disadvantage that the concept of interaction with a computer 
cannot be fully relayed to a novice user via the printed medium. Such points 
have been discussed in greater detail in the survey of literature, chapter 2. 

On a more specific level, ccxnments offered by all participants in the 
three experimental groups have been categori.zed and will be presented along 
with certain suggestions for improving the format and contents of the User 
Guide . 

A. General Layout 

fflie reader is referred to Appendix A for the format of the User Guide 
in question. It is noted that page one of the guide is very important but 
rather misleading to the non-specialist use : of the terminal. He may feel. 
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and Justifiably so, that he is not "familiar with the essential features of 
the Information Bank;" at which point he is left with the message that he must 
read everything in the manual, ihis means that he must read eight pages of 
Information, including the mechanics and maintenance of the printer, before 
getting to a sample search on the Information Bank starting on page 10. 

Experience with the feasibility group shewed that by the time the 
user had read the first few pages of the Guide , he had totally lost inter- 
est r patience with this procedure. In many cases he had not even reached 
page 10 before he decided to try a search on the terminal. Because of this 
reaction, the participants in the formal experiment were guided to page 10 
"A Basic Information Bank Search" with the aid of a paper clip and a brief 
prompting by the experimenter on the general layout of the User Guide . The 
results of this procedural change are qiiite evident by the rise in the success 
rate of the non-tutorial group in the formal experiment. 

In a recent communication with the Information Bank staff, the author 
was informed that the new User Guide will no longer include information on 
hardware description and maintenance procedures. Because of the V2uriety of 
terminals introduced in remote locations, a separate manual will be produced 
to deal with the idiosyncrasies of Individual terminals and printers. 

B* "Ihe Sin>, • .arch" and "What You Were Doing and Wliy" 

It was suggested by a number of users that they would have had a much 
better understanding of the simple search if the reasons tor their response 
were identified either before or during the search rather than after (see 
pp. 10-14 of the User Guide ). 
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If the present layout of the manual were to be changed to accommodate 
a colximn reserved for explanation of suggested responses, thus interjecting 
comments made on page 14 into the previous four pages, this problem may be 
resolved* 

C. "Do This First" 

This heading refers to a sample search portrayed in the User Guide 
(pages 10-12 )• As this was the nxist-used portion of the manual, it elicited 
the greatest number of comments by che users* 

1* System messages as well as user responses are printed in green to 
simulate the Information Bank "screen." Many individuals do not differentiate 
between the two and assume that all material printed in green refers to system 
messages. This triggers the most frequently heard complaint about the sample 
search: "Why are the instructions printed backwards (or ups:*'^'^ down)?!" This 
question refers to the ambiguity encountered when matching Tiysccm messages with 
their corresponding suggested responses. For a clearer understanding of this 
error a portion of tnese instructions have been reproduced in Figure 7.1. The 
University of Pittsburgh identification number does not require a password; 
thus, screen instructions skip from #1 to #3. The user looks at this screen 
and looks for a matching message in the User Guide . He keeps reading on; 
assuming that Instruction #4 refers to System Message #3. The majority of 
users look at the system message and read down rather than up to find the 
suggested response. When the User Guide states '^The next step will look 
like this," the user naturally looks for the closest message to the sentence. 
By doing so, he is in fact looking at the prior system message. Wherever this 
error occurs, the stage is set for the user to be one step ahead of the system 
message on-line. Users who depend upon the Guide for instructions tend to 
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Do this first 



Step I -Sign-on 

• When you sec this message type 
in your identiriecilion number 
thenPRHSS Till* I N THR KL'Y 

Tt r ft I n a ( cp* n 

• Next type in your password 

(if you arc at a terminal whieh aski 
for this second siim-on step) 

AND i>Kt':ss i-nii:r 

Y«ur p«*iward >■ r«quir«ti *,\ * Itfrnifitl 

CHTCR 0//p»nw»r<J 
t nf tb Ank 

• The next step will look lik^* 

this. Type: to selcel the s[>eiied- 
out system message and I'Rl-.SS 

enti;r 



/b 



^•■a afaa "6 



Instxuctlon #1 

System Message #1 
Response if\ 

Instruction 

System Message #2 
Response ^2 

Instruction #3 

System Message #3 
Response il3 



Instruction fM 



At this point you will see a 
message from tlie system mon- 
itor, usually a list of data base 
contents. Type: .t to proceed and 
PRHSS liNTl^R 

nOTfi OPSf CONTnlNS NOV I- 15 I 969 . I 070 . I 97 I . I O 7 2 OE! C l^. nMO ft 
C NOH TiMes Hi^TfPini. FULL TEXT PVPIL SVSTF.H M(iHITOR 

[f yeg wtah to 
p Procaait. lN7fR « 

e S«ntt n«aa4lB to Sytt«n Monitor b«for« prccvaftinS* LNTfP b 



System Message #4 
Response #4 



c 1973 

Reproduced by Permission of The New York Times 

Figure ?•! 

A portion of a *^Basic Search'^ as it appears in the Information 
Bank User Guide 
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to follow the printed instructions rather than the ones they receive on-line • 
They go so far as to enter conroands which do not even appear on-line simply 
because they are following the wrong set of instructions from the User Guide # 

Many users do not realize that the characters printed after the pound 
sign (jC) are what they should be typing in response to a message. Some have 
suggested the use of an arrow or so me o ther form of visual clue to show the 
user what he will be typing at every step. 

The example used in the search has also proved confusing to some. 
Since the same term (Wankel) has been used with three separate meanings, it 
nurtures the thought that various concepts in the user's mind can be linked 
together at the term entry and term selection stages. Some novice users lose 
sight of the "alphabetic" nearness of the terms used in the example and expect 
to see all of their own input appear together on the screen of the Information 
.Bank terminal. 

D. Additional Suggestions 

It may prove useful to link each step of the "Sample Search" to its 
corresponding section in the comprehensive User Guide by a page reference 
number. This would allow the user needing more detailed information to go 
directly to the step in question in the larger manual. 

Since the modificaticDn^'stage is skipped in the simple search, the 
user needing to modify an inquiry should be referred (by a page number) to 
the section where modification procedures are explained (p. 18). Such simple 
referexKTing procedures may save the user much time and effort in paging 
through juides for a single direction. 
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It is also suggested that at least the major universal options of "X" 
and ••2" be explained to the user as a part of the basic search. 

Even though the User Guide in question is only 32 pages long, it is 
very much in need of a thorough index for easy referral to any portion of its 
contents* 
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CHAPTER VIII 

THE NEW YORK TIKES INFORf-tATION BANK USER/SYSTEM INTERFACE 

Apart from the quantitative findings of the experiment, this study 
yielded extensive qualitative results, esp>ecially in the area of protocol 
analysis. In fact, the subjective analysis of user reactions to the system 
in general, and specific stages of a search in particular, provides a rich 
source of coamentary on the effectiveness of system messages in guiding a 
flrst--time user through a successful seeoxh. Such comments, of necessity, 
reflect the reactions of the academic community to the Information Bank. 
Although much c the discussion in the ensuing pages may be generalized to 
other user groups, some comments specifically reflect the thought processes 
of the student population. 

In his recent analysis of eleven interactive retrieval systems Thomas 
Martin notes "It rapidly became clear that designers knew little about the 
habits and characteristics of individual users. . . .Only gross generaliza- 
tions were available regarding whether end users were carrying out their own 
searches." U) The Information Bank is no exception in this respect. A 1973 
article by John Rothman, Director of Information Services of The New York Times, 
seems to identify the newspaper reporter and staff as the major user population: 

The messages that form the computer's part of this 
dialogue are in terse but conversational, non- technical 
English. Ihese messages and the diverse options and 
instructions that they presisnt were desicned with the 
newspaper reporter and editor in mind. Emphasis mine (2^) 
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This is a very fair and Justifiable statement and certainly seems to 
give the designer a specific lead on the ultimate user population of his sys- 
tem. However, in a more recent communication, Dr. Rothman clarifies the above 
statement as follows: 

This (that the system was "designed with the newspaper 
reporter and editor in mind 'J is misleading as stated. The 
system was intended from the very beginning to serve a variety 
of outside users — business, government, educational institu- 
tions, and the like~aG well as our newsmen; however, the 
vocabulary and the specifics of the inquiry process are based 
largely on what we )cnew and learned from the uses of The Times 
Index and Morgue. V/e were very well aware of the fact that our 
user population would include a Icirge number of casual and tram- 
sient users who would often be in a hurry and many of whom would 
be tinskilled in research techniques and unfamiliar with inter- 
active computerized retrieval systems. (3,) 

It is quite justifiable, and in fact desireable, to design and build a 
system with a specific user population in mind. However, it is a more diffi- 
oilt task to broaden the scope of the market for a bibliographic information 
retrieval system to a point where the end users may have conflicting, or at 
least unrelated, information needs. Although it is obvious that a broad seg- 
ment of the population reads the daily newspaper, there is no evidence to show 
that all readers peruse this medium in the same manner or for the same purpose. 
Indeed, it would be contrary to conmon sense to claim such a thing. By the 
same token it cannot be reasonably expected that an abstracted version of the 
news, made available through a single controlled vocabulary and common commands 
can answer the needs of such a varied population. While the journalist may 
find easy access to the files through personal or organization names, the 
academician may prefer a finer breakdown of the news by subject. While an 
editor may be looking for clear facts, the student is still groping for ideo- 
logical or philosophical concepts related to world affairs. Wheh we discover 
that it is virtually impossible to track down certain newspaper articles 
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dealing with a concept undefineable through the thesaurus, it becomes ques- 
tionable that the designers of the system had fully identified the ultimate 
user population. In describing the aforementioned report by Martin, he 
states : 

..r.::t. ' Missing from what follows is a clear picture of the 

users, how frequently they search, and the problems they 
run into. System representatives were asked to charac- 
terize their users but few could do so. U) 

Taking this juxtaposition a step further, Dr# Rothman believes the 
Information Bank to be "largely self-teaching and self-service" and asserts 
that "the interposition of librarians or information specialists is not 
required." (5) Assuming the usability of the Infoxnnation Bank by a 'Variety 
of outside users," accepting the self- teaching capabilities of the system, 
and limiting the discussion to the academic community, the system interface 
will be analyzed on three levels. 

Firsts some general comments will be made regarding the feature or 
coninands which apply to more tham one individual "screen" or step. Next, 
there will be a screen-by-screen emalysis of instructional messages as well 
as some conceptual problems faced by users at specific steps» Finally, there 
will be an overview of the system as it compares to "minimal design feattires" 
recommended by the eleven system designers participating in the Stanford 
Workshop. At each level of analysis the author will also present certain 
recommendations based upon user comments offered in the cotarse of the past 
two years, as well as results of the experiment and the overall experience 
gained in training novice users of the system. All three levels of analysis 
are arranged, whenever possible, to follow the flow of a complete search 
strategy; only the full form of systems messages (recommended for the novice 
usar) has been taken into consideration. 
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A final note on the ensuing recommendations: most changes may be im- 
plemented with a minimum of re-programming and cost* However, at first glance, 
some of these changes may seem upsetting to the present users of the sys.':Gm who 
have ••memorized" all options and corimands. On the other hand, keeping tlie; ex- 
panding user population in mind, the longer such char.ges are put off, the harder 
they will be to implement as the number of users wishing to keep the status quo 
grows. In looking back at some of the changes which have hecA made since 
November of 1972, we can easily see the long-range positive effects far out- 
weigh some of the immediate hardship involved. 

In cases where 'major additions or changes are recommended, it is done 
so with the understanding that such revision may not be feasible in the near 
future. However, the author feels it is important to bring to light those 
features which are needed to make the system more responsive to human needs. 
These user-oriented principles •'will not be backed up with citations to a well- 
developed body of man-computer problem-solving literature because that body of 
evidence has not yet been created. •' (jS) 

I. General Comments 

Most designers of on-line systems consider a user manual and live 
instructions as essential elements of a system. This fact was brought to 
light during the Stanford V/orkshop, but Martin adds ''One should not conclude, 
however, that on-line documentation and assistance are unimportant." (2) Ihis 
author contends that supplementary on-line instruction is absolutely essential 
when a transient user population is expected. 

On-line instructions can be kept up-to-date vrtiile printing and distrib- 
uting new mcinuals may be more time consuming and costly. If printing costs eure 
to be cut down by revising certain pages 'of the manual, it is difficult to 
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monitor all remote stations and make sure such revisions are recorded properly. 
A survey by the National Library of Medicine showed that the user manuals were 
not Jcept up-to-<late or actively used by the subscribers. In many instances 
such manuals were not even kept close to the terminal location. (8) Such 
problems could be avoided by keeping a fully updated version of all system 
instructions on-line. 

However, it is not enough to convert the printed user guide into 
machine readable form and provide a simple index to the text. Time and time 
again it has been shown that ••users learn by doing." (9) It is a pity that the 
majority of present day on-line systems do not take advantage of their inter- 
active capabilities to communicate the essence of an interactive search to the 
novice user. Even where instructional texts are available on-line (such as 
^ilDLINE) little attempt is made to test the user*s tanderstanding of what he 
has Just read. Interaction, even at its simplest level of drill-and-practice, 
can be a great boon to the first time user of the system. 

Informative notes, apart from a complete tutorial, may be a first step 
toward creating a total system which would be truly user-oriented, allowing for 
full intieractlon without the "interposition of librarians or information special 
ists." The next section will address some of these needs as applied to specific 
Instructional messages of a particular screen. The following recommendations 
apply to the system in general or at least refer to topics related to two or 
more screens. 

1. Immediately upon "sign-on" the user should be given the option to 
view "explanatory notes." Such a step, in the futiare, may lead the user to an 
on-line tutorial program. For the present, it can perform the following fxinc- 
tions: 
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A. Upon specific demand, show the user a list of "new terms" 
added to the file* Based upon file maintenance procedures, this 
list may be changed weekly, bi-weekly, or monthly. In any case, 
this would be of great advantage to the novice and well as the 
experienced user. New terms are constantly added to the file, 
but they are immediately lost to the alphabetic order of a very 
large thesaurus. 

It was announced in November 1972 that a new index term has 
been added to the file. "New search terms" is a most recent addi- 
tion to the thesaurus which answers the basic need mentioned above. 
However, as is presently implemented, the perusal of such a list 
requires the user to complete a full search and view these new 
terms as he would the abstracts to articles; i.e., as the final 
output. For those subscribers who are being charged on the basis 
of connect- time this feature implies the payment for a full search 
Just to rind out what terms are "searchable." Ihis arrangement 
does not seem satisfactory to the author. New search terms should 
be listed in alphabetic order and the system should provide a means 
for selecting appropriate Ivrns a!: once, at the beginning of the 
session. 

B. Upon demand, give a more elaborate description of the current 
status of the data base than what appears on the third screen in 
the present format. A short description of non-Times sources 
would be particularly informative to the users. 

C* Upon demand, describe the universal options of the system. 
These options which are in fact essential ingredients of a search 
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(such as the universal option to terminate a search) are 
presently lost in the verbiage of the User Guide > Their 
function is described on page 31 of the Guide in the glossary 
of terms. One wonders hew a first*time user is expected to 
come upon this definition if he is not aware of the existence 
of •^Jniversal keys." 

D« On a temporary basis, until such time that a tutorial is 
designed, it would be helpful to have a one-page (one screen) 
explanation of the flow of steps in a normal se«trch« This may 
be similar to the "blue card" (Appendix B) designed to familiar- 
ize the participzmts in this study's experiment with the stages 
of an Information Bank search. As the results of the experiment 
confirmed, many novice users have difficulty conceptualizing the 
flow of search steps. It would be to the advantage of this group 
to get acquainted with the system through this suggested option* 
Following the general format of instructional messages pres- 
ently offered by the Information Bank, all above options may be 
presented in menu form. In order that such an additional step 
would not hinder the search process for the experienced user 
(who may not be looking for new term entries), the screen imme- 
diately after sign-on may be ammended as follows: 

Use abbreviated forms of system messages «= A. 

Use fxall forms of ff/stem messages = 3. 

View explanatory notes (search steps, new terms, etc.) = 

2. Revision and expansion of error messages. Most of the present 

error messages are of no value to the user beyond informing him that whatever 

response he had entered was unacceptable to the system. In the few cases where 
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explanations are offered-^ith the exception of date modification error messa- 
ges—they prove to be more confusing than helpful* Error messages specific to 
a given screen will be discussed in their appropriate context in the next sec- 
tion. The following comments are applicable to more than one screen. 

A. The system's universal error message is "INCORRECT RESPONSE 
TO MESSAGE. PLEASE RE-READ INSTRUCTIONS". The most helpful 
addition to this message would b5 to display the user response 
which triggered its display. That is, whenever an error message* 
appears, the system must repeat the user's last response. Such 

a display would in itself solve many misunderstandings caused. by 
the user's carelessness in spelling or typing. In the absence 
of such prompting, and when the user thinks that he had made the 
correct response, he may be forced to choose a wrong option be- 
cause he has no access to his last response which might have 
given him a clue as to where he went wrong. 

B. The overwhelming user diffic\ilty in interpreting the phrase 
•View continuation" has been mentioned in previous chapters. It 
is a phrase used often throughout the instructional messages of 
the Information Bank. Here, in free from, are a representative 
sample of definitions offered by users for the phrase "view 
continuation": 

"It means to go on and see the abstracts." 
"Continue to look for my other terms." 

"Continue to look at the list until I find the combination 
of my terms." 

.jntinue with my se.xrch." 
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In juxtaposition with the next option which always allows 
the user to "go to next step", some users felt that they must 
choose to "view continuation": 

"But I haven't chosen all of my terms yet? I am not ready 
for the next step." 

"I am not ready to look at the abstracts yet." 

Thus we see that Just as some users felt obliged to view 
the continuation because of misinterpreting its meaning, others 
chose this option by default, because they were misinterpreting 
the meaning of "go to next step". Add to these the numerous 
users who admitted total confusion ("I'm completely stuck, I 
have no idea what it means"), and we have the opinion of the 
majority of users on this topic. 

Although the nature of this statement makes it impossible 
to monitor the motives of the user when he chooses to 'View 
continuation", it would seem very reasonable to provide a 
minimum of guideline for those who are obviously misusing 
the option. 

It is understood by the experienced user that whenever an 
"End of Display" sign appears at the end of a page, there is 
no continuation to that screen. It is suggested that this 
simple test be monitored by the system and whenever a user 
chooses an option which implies 'View continuation" from a 
screen containing the "End of Display" message, the following 
error message be offered: 

107 

95 



**lhere ia no continuation to this screen* Choose snother 
option. 

Presently the system slpoly keeps repeating the same screen 
offering no indication to the bewildered user as to what has 
caused such repetition. 

Another means of lessening the user's confusion would be to 
clarify the phrase perhaps by stating "view continuation of 
present list." 

C. A universal error, especially applicable to experienced 
typists, is to use the lower case of the letter instead 
of the numeral "1**. It is suggested that whenever the system 
is e:cpecting numeric input only that a test be made to detect 
this error. It seems only reasonable to make allowances for 
this type of human error wherever posfsible. 

0. At times, when an extra space is inserted in a response, 
a word such as "and" is typed between two term numbers at the 
term selection stage*, or other similar mistakes are made, the 
error message reads: 

"A VALUE ENTERED WAS NOT PRESENT IN THE LIST OF ITEMS 
DISPLAYED*" 

Many users unfamiliar with 'computer language" are unable 
to extend the definition of "value" to non-numeric elements 
or chcuracters. Thus, they are rrcre confused than assisted by 
this message. Perhaps, if a more distinct message cannot be 
offered, it would be best to replace this with the standard 
error message of the system. 
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3» Ihe universal key "W" provides an explanation of abbreviations used 
in the Information Bank. This option has great potentials which have not been 
exploited as yet. As it stands, the explanations offered are nothing more than 
the full spelling of the abbreviated items. It does not define purpose or pro- 
cedures for using the abbreviations. In, more cases than not, when the user 
types "W" he r-counters the catch-all phrase "NO EXPLANATION MESSAGE AVAILABLE". 

It is suggested that without going into a deep discussion of options or 
abbreviations, a short but uniform amount of explanation be offered for key 
terms and abbreviations appearing in all instructional messages. A very good 
example of an instruction requiring explanation for non- journalist users is 
the modification stage. Although details of problems will be discussed in 
the next section, it may be illustrated here that "sketch", "type of material", 
or "source" each have totally different meanings outside of journalistic 
circles. ' 

4. In connection with the system error messages it was suggested that 
the last user response be repeated on the following screen. This is presently 
done only on the second page of the logic stage. It may be pointed out that 
such an option may be used for a variety of reasons, other than error detec- 
tion, throughout the search. It basically "confirms" the user response and 
may be especially useful dioring modification and logic stages. 

5. Current system programming and Incoterm hardware do not allow for 
a differentiation between a legitimate time delay and total system failure. 
In either case a light marked "system not available" comes on to denote that 
the user has lost contact with the computer. Also, many novice users do not 
associate the light on the keyboard marked "enter pending" with the fact that 
their latest response has been — or is being — relayed to the computer. Thus, 
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whenever an Immediate response is not forthcoming, the user becomes very anxi- 
ous and repeats pressing the "Enter" key and otherwise displaying his concern. 

With the knowledge that The New York Times Information Bank is currently 
using a variety of hardware at its remote locations it . is suggested that a pro- 
gramming addition be made to give verbal (printed) or visual clues to the user 
as to whether it is his turn or the computer's turn to work on the query. 

The remaining suggestions are not "informative" to the user in the 
same sense as those offered above. Nevertheless they require careful consider^ 
ation by the Information Bank staff as they affect the ease with which a user 
carries on a normal seairch. 

6. In all cases where the first two options appearing on the screen 
are the same— except the first option allows viewing of continuation — it is 
suggested that the order of these options be reversed. 

In the majority of normal searches the user does not need to view 
continuation. But we have already observed that most everyone reads instruc- 
tional messages from top of the screen and stops reading as soon as he finds 
an option which seems to satisfy his basic needs. Thus, m.any users never 
reach the second option which is really best suited to their needs. 

It is also very important to note that in many cases, such as term 
entry from the title abbreviation screen and many of the modification steps, 
the first option is currently unnecessary and cannot be used by anyone for any 
purpose. By appearing at the top of the screen, such options help no one and 
can only harm thfe first time user by making his -^aarch process more tedious. 

7. In keeping with the overall uniformity of system messages and 
preserving a sense of identification with each option, it is suggested that 
at the point of viewing abstracts the "C" option be allowed to carry out its 
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established meaning of 'View continuation. " At present this option terminates 
the query at hand. Many users who have gone through the entire search as.'soci- 
ating the "C" option with "continuation" automatically type "C" when they wish 
to view the next page of abstracts, thus unintentionally terminating their 
search. 

8. There is a definite need for a fail-safe step which would warn the 
user that his current search is about to be cancelled. 

The universal option of "Z" terminates a search at any given point. 
It is not unusual for an individual to press this key by mistake, especially . 
If he had intended to type "A" and his finger had slipped to the next row on 
the keyboard. Also, as mentioned in item number 7 above, the user may have 
typed "C" rather than ''A" to view the continuation of abstracts, ^is action 
would lead him to the following message: 

A. Begin another inquiry, ENTER: A 

B. Terminate, ENTER: Z 

For those unfamiliar with the universal option "X" which allows the 
user to go back to the previous major step (in this case the logic step), 
there seems to be no way of recovering the search they had unintentionally 
dropped. 

It would seem fair to give all users a second chance to decide the 
fate of their current search. Instead of simply erasing all transactions as 
soon as the "Z" option is requested, the system might "hold'* the query for 
further verification of the termination request. In cases where pressing the 
"Z" key has been uni.ntentional , such a fail-safe step can save much time and 
effort without unduly affecting normal search processes. 



99 
111 



II. Screen-by^'screen Analysis of the Information Bank 
km Slgn-on 

Terminal Open 

ENTER A/ /identification 

The above message greets the novice user as he approaches the termi- 
nal. .Although it looks self--explanatory, a majority of first-time users 
Ignore the "A//" format. Most proceed to type the identification number with- 
out a format and are immediately confronted with an error message. But there 
2Lre tnose who feel that they can enter their query at this point (defining 
identification as **identif ication of search terms 1") It has been the experi- 
ence of this author that in such cases the system response is very slow. Soi;te- 
times there is no response unless the screen is "reset" and the corx^^^- identi- 
fication entered. 

Returning to the group who entered the identification number without a 
format, they are now faced with the following error message: 

INCORRECT RESPONSE TO MESSAGE. PLEASE RE-READ INSTRUCTIONS 
OJIT = A 

ELAPSED TIME = 0 MINUTES 

Of over thirty participants in the experiment confronted with this 
screen, only one was able to correctly interpret this message and return to 
the identification entry step. Sorne of the representative comments about this 
error message are as follows: 
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"How can I re-read instructions if there are none offered?" 

"•How can I quit if I haven* t even started?" 

"Does Quit «= A mean that I should quit using "A"?" 

One participant interpreted this message to mean that an identifica- 
tion number was not required (since he had typed it and it was rejected) and 
proceeded to enter his query terms. Another participant, after making the 
same formatting error on three occasions and being told to "follow the instruc- 
tions literally" proceeded to type: 

A// identification 

instead of entering the identification number* ' 

The results of this experience were discussed with the Director of 
Information Services of The New York Times. It was stated that this step has 
been made intentionally difficult to discourage unauthorized use of the system. 
But it must be assumed that only authorized individuals have access to the 
actual identification number and password (which is required at some locations). 
There is also no question of privacy or security involved in this data base as 
far as subscribers are concerned. In addition, there is no danger of input or 
file manipulation from remote locations using subscriber identification numbers. 
As a final point, we will see that a more reasonable error message is already 
available to those who type the correct format but the wrong identification 
number. Since it is the identification number and not the format which dis- 
tinguishes "authorized" from "unauthorized" users, it seems rather harsh to 
punish the novice user for forgetting to follow the acceptable format by 
denying him access to the instructional message at the point of sign-on. 

113 



101 



If the user had typed the correct format but the wrong identification* 
which is just as likely to happen to an unauthorized user~he would be given 
the following error message: 

, IDENTIFICATION ENTERED IS UI>2CN0WN TO THE SYSTEM 
CORRECT IDENTIFICATION REQUIRED. EI4TER A// IDENTIFICATION 

This message is repeated twice before the user is reJFerred to his instructor 
for assistance. This error message seems more reasonable and it is suggested 
that it be used for all user errors at the identification stage. 
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B« System Messages 

Use abbreviated forms of system messages = A. 
Use full forms of system messages - B. 

In a normal search — ^barring mistakes made during identification entry — this 
is the second screen the user encounters. For the novice user who needs full 
instructions, this is the only screen with a totally different format. If the 
same instructions were to be re~written to match all other screens in the 
"full forms" format, this is the way it would look: 
If you wish to: 

A* Use abbreviated forms of system messages, ENTER: A 
B. Use full forms of system messages, ENTER: B 

Instead, the present format conforms with the "abbreviated forms of system 
messages." Although the User Guide suggests that all punctuation marks in 
instructional messages be ignored, still many users are misled by such extra 
characters. It would be helpful to present this screen in the "full forms" 
version. 

However, the important issue at this step is the content rather than 
the foxrm. "System messages" has no meaning to most users unfamiliar with 
computer jargon. Here is a sample of personal interpretations of this 
message: 

"If I want to abbreviate I use A; if I want to spell out my words I 

use B." 

"I send my message here?" 
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After choosing the "A" option: "1 just want to take a quick look at 
what they have." 

. Another reason for choosing the "A" option: "I thought it would be 
easier; I have a short search. " 

Others chose the "A" option simply because it appeared first on the 

scrc*en. 

It would be a sL-nple task to replace the phrase "system messages" 
with more familiar terminology such as "Instructional messages" or simply 
"instructions." 
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C. Messages From the Monitor (System Bulletin) 



(Data base contents and other nessages from the Monitor) 
If you wish to: 

A. Proceed, ENTER: A 

B. Send message to System Monitor before proceeding, ENTER: B 

The monitor messages change from day to day or even a number of times during 
a single day. The questions posed by the novice user at this point usually 
concern the monitor himself or the terminology he uses in his messages. For 
example, the meaning of "FTV" (full text viewing) or "deferred printing" are 
questioned. Those individuals arocious to begin the search process sometimes 
type the letter "A" to proceed and follow that by their search terms. 

Generally, there is little problem encountered by the novice user at 
this step. 
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D. Term Entry 



Enter (or add more) search terms* 

A. Enter 1 or more search terms (separated by //) , 
ENTER : A/ /term/ /term// . . .etc. 

B. View title abbreviations then enter search terms, 
ENTER: B 

By this time the user has become familiar with formatting procedures. 
In cases where the problem still occuis the user is able to detect the error 
and correct the format. 

Based upon thesaural errors made during the term entry process, a 
short guide was develop>ed to complement the second CAI session dealing with 
term entry and selection. This guide appears in Appendix B and may be 
referred to if the reader wishes to identify the major areas where novice 
users needed assistance in the term entry process. It would be beyond the 
scope of the present study to address such problems in detail. 

Other than the conceptual problems involved in selecting appropriate 
index terms, the basic problem at this step was the "B" option. One user 
complained that since he had asked for the "full forms of system messages" 
he should not be getting "abbreviation*/ of titles. 

The most common interpretation for "title abbreviations" is the 
abbreviation to titles of articles (rather than personal names). Since 
many existing bibliographic information retrieval systems contain an option 
to search by title of eurticle or publication, the novice user extends such 
meaning to the Information Bank data base structure. His problem becomes 
more acute when he actually requests to 'View title abbreviations." He is 
then confronted with the following list: 
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Enter (or add more) search ter/ns. 

A. Enter 1 or wore terms and view continuation. 
ENTER: A// term/ /term/ / . . . etc. 

B. Enter 1 or more terms and go to next step. 
ENTER : B// term// term// . . .etc. 

C. View continuation, ENTER: C 

D. Go to next step, ENTER: D 

PRS MYR GOV SEN REP MRS VP SIR CLM ADR ADM AMP 
ABP ADK ASM AAG ALI ADN ADI ADA APG ASC ASG ASU 

SP4 SP5 SP6 SP7 SP8 SP9 SEP SUP SPV SUG SUR IRS 
use USG USA VRV VAD VCH VCL VMA VMI VIS COS 

* * *END OF DISPLAY* * * 

If he is the typical first tiine user, he does not know about the •'W" 
option to view 'che full form of such abbreviations. Thus, after a moment of 
staring at the list of abbreviations he tries to continue his search by dis- 
regarding it. He then proceeds to read his instructions from the top of the 
screen and chooses the "A" option to enter his terms. Upon pressing the 
"Enter" key, this screen repeats itself — the "End of Display" sign indicat- 
ing that there is no continuation to this screen* Totally bewildered by this 
"magic" disappearance of his terms and the repetition of the same screen, the 
user now proceeds to re-^nter the same terms. If he is curious, he may try 
the "B" option the second time; if not, he may use the "A" option again, 
switching over to "B" only after several tries fail to take him past this 
step. Meanwhile, the computer has been systematically saving all of the 
user's entries. After the command of "go to next step" is finally given 
through the "B" or the "D" option, all entries will be displayed to the 
user on a sequential basis. It is not difficult to imagine the user's con- 
cern and frustration when he must repeatedly decide to accept or reject the 
saune term. 
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Biis is one of the cases mentioned earlier where the "A** option is 
of no value to the user (because of the "End of Display" sign) but causes 
. much frustration by simply appearing first on the list of options available. 
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E. Term Not in Pile 



This term is not in the file. If you wish to: 

A. Enter replacement term, ENTER: A//term 

B. View title abbreviations and enter replacement term, ENTER: B 

C. Reject term, ENTER: C 

D. Suggest this term be added to the file, ENTER: D 
(term in question appears here) 



There were many objections to this screen. Because the term was 
placed immediately after the "D" option, many users failed to notice it and 
then wondered which term the system was referring to. Some felt that all of 
their terms~or the one picked from a previous screen— had been rejected. 
There was no indication given as to why the term was not in the file or what^, 
would be done to rectify the problem. An early report to the Information 
Bank staff resulted in the subsequent changing of that screen to the following: 



The term below is not in the file. Have you made a spelling 
or format error? If you wish to 

A. Enter the term again spelled differently ''^(or another term) 
ENTER: A/ /term 

B. View title abbreviations and enter term, ENTER: B 

C. Reject term, ENTER: C 

D. Suggest this term be added to file, ENTER: D 
(The term in question now appears here) 



Ihe present format solves all of the previous problems. But one 
more hurdle still exists. When the user finds that his term is not in the 
file he decides to view title abbreviations as a last resort. In two years 
of observing academic users of this system, the author has not encountered 
a single case where title abbreviations were used successfully by a non- 
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specialist user. As described earlier, "titles" are normally defined as 
"titles of articles" by the academic user* It may also be added that what 
the academic community really needs is a means of modifying a geographic 
location by a title, to find the name of the president or prime minister of 
a country or the name of the mayor of a city. If the user already knows the 
name, he has no need to modify by title. Most cases of doubts concerning the 
correct spelling of the name are rectified through means other than the use 
of title abbreviations. 
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p. Inversion of Terms 



Try an inversion of this term. 

A. Enter term with changed word order, ENTER: A/ /term 

B. Reject term, ENTER: B 

UNITED FEDERATION OF TEACHERS 

The instructions on this screen are quite simple amd straightforward 
But two facts remain to be clarified: 

1« How do you invert a complex term? In this case, is it "Teachers 
United Federation of", or "Federation of teachers. United"? There are many 
more complex examples which may be cited. 

2. The. system seems to infer that the inverted form of the term in 
question is actually in the file. A user entering a complex term through a 
variety of inversions may finally realize that regardless of his form of - 
input the system cannot accept his term. 

It is suggested that the caption to this screen be modified as 

follows : 

The inverted form of this term may be in the file. 

The system should also offer a minixnum amount of help by prompting 
the user to "be store to enter a space after the comma," as this is a common 
cause for rejection of inverted terms. 



Ill 



123 



G<» Term Selection From a Menu 



If you wish to: 

A. Pick term(s) by number from list below and view continuation 
ENTER: A/ /term number/ /term number. . .etc. 

B. Pick term(s) by number from list below and go to next step. 
ENTER: B//term number// term number'. . .etc. 

C. View continuation. ENTER: C 

D. View information (XRF: notes, cross refs) about 1 term 
Enter: D//term number 

E. Go to the next step. ' ENTER: E 

1 AUTOMOBILES 

SUB XRF 8192'16383/C 

2 AUTOMOBILES SALES AND SALESMEN 

SUB 1/C 

* * *END OF DISPLAY* * * 



Certain general comments are in order before examining individual 
option problems. 

1. It has been observed that many students analyze this screen as 
though it represents an "outline". From their point of view all numeric 
divisions fall under the alphabetic division of "E". Thus, they are con- 
vinced that numbers must be picked in relation to the letter Many such 
individuals respond with •*E//l//2, etc." or "B//E1" when they should be typing 
"B//1//2, etc." 

2. Most individuals do not differentiate between "term number" and 
the "citation range." 

In order to alleviate both the "outline" problem and the confusion 
related to term numbers, it is suggested that a heading be Inserted above the 
colLDTins representing term numbers and the citation range. So that the experi- 
enced user would not be deprived of receivir ^ an extra term per screen (the 
line taken up by such a heading), headings may be attached to the full forms 
of system messages only. 124 
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On a more specific level, the following observations are noteworthy: 

1. Hie "view continuation" as well as the "A" vs. "B" problems are 
quite obvious at this stage. Again, it is urged that an error message be 
offered if the user attempts to pick the "A" or the "C/ option from a screen 
showing an "End of Display" sign. Also a slight modification of the phrase 
•View continuation" — would help the user who may be confused about the mean- 
ing of the option. 

2. Most novice users of the system do not understand the significance 
of the "XRF" (Cross-reference) attached to a thesaurus term. These individuals 
request to "viev; information" about a term not tagged with "XRF", using the "D" 
option. The error message at this point informs the user; 

" THIS TERM HAS NO NOTES OR CROSS- REFERENCES 

It is strongly recommended that the wording of this error message be 
made positive and more informative. For example: 

INFORMATION IS AVAILABLE FOR TERMS TAGGED WITH XRF ONLY 

3. In reference to the citation range, it must be noted that the 
majority of users do not become aware of its significance on the first search 
unless it is brought to their attention by a human assistant. If these figures 
were identified by a heading, as suggested above, this problem would be solved 
to a certain extent. It would also be much less confusing if instead of show- 
ing a binary range, which is unfamiliar to many, the actual number of postings 
were to be shown. 
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«(• Accepting Terms 



If you wish to: 

A. Accept the term as given below, ENTER: A 

B. Reject this term, ENTER: B 

C. See more information about this term, ENTER: C 

JEWELS AND JEWELRY 

SUB XRF "c. 256-511/C 

Although this screen is by far easier to understand than the one dis- 
cussed above, it still presents its own unique problems. 

1. Unfortunately, the sample search presented in the User Guide does 
not explain this screen. Those who are depending entirely upon the instruc- 
tions offered in the Guide see an tinfamiliar screen for the first time at this 
point. The roost normal reaction is to ask for '^ore information". By doing 
so, the user is faced with yet another unfamiliar screen. F-any first-time 
users are unaible to go beyond the cross-reference screen without human assis- 
tance. 

2. Ihe major problem with this screen is not so much instructional 
as it is conceptual. When a user has selected a term from the menu (G above) 
vrfiich contains an automatic cross-reference, he must again decide to accept 
or reject the term he has already picked after viewing the cross-reference 
screen. 

Most novice users are simply confused as to why they must pick the 
same term twice . Some proceed to reject the term on the second round to 
avoid redundancy. 

i 
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As an experienced user, the author is in full sympathy with the 
novice* It is frustrating enough to be subjected to an automatic cross- 
reference which is usually of no assistance, but to have to r€K:onsider the 
acceptance of a term already chosen is indeed unjustified. Of course, the 
author understands the significance of such a step in warning the user of 
certain scope limitations of the term he has chosen, or familiarizing him 
with related terms. But in every case where the user needed such information 
or explanation the chances are quite high that he had already taken advantage 
of the "D" option to look at the cross-reference before selecting the term. 
This means that the conscientious user is in fact forced to look twice at the 
cross-reference (once before, and automatically once after term selection) as 
well as having to select his term twice (once from the menu ax^d a second time 
after he has viewed the automatic cross-reference, ) thus going through four 
steps just to pick a single term. This is indeed a high price to pay in order 
to avoid having an "unwanted" term at the logic stage. 

It is an established fact that all terms chosen by a user are displayed 
at the logic stage. Assuming that an "unwanted" term appears at that stage, it 
can easily be ignored and it can do absolutely no harm to either the user or 
the system. 

It must be noted that there is a "precision" key which allows the user 
who knows he is asking for a legal index term to skip the thesaural display 
stage. However, even this shortcut is not immune from the two extra steps of 
viewing an automatic cross-reference and re-accepting the term entered with 
the precision character. 
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!• Cross*-reference 



If you wish to: 

A. Pick term(s) by number from list below and view continuation. 
ENTER: A/ / term number/ /term number. . .etc. 

B. Pick term(s) by number from list below and go to next step. 
ENTER: B//term number/ /term number. . .etc. 

C. View continuation, ENTER: C 

D. Go to the next step, ENTER: D 

ACCIDENTS AND SAFETY 

SUB • 1969-1969 XRF lOlA-lOAl/C 

NTE In regard to means of transportation, material on 
NTE delays, whether or not due to accidents, is 
NTE included. 

1 SAL NT SUB ASPHYXIATION AND SUFFOCATION 

2 SAL NT SUB BURNS 

10 SAL RT SUB LIFE-SAVING EQUIPMENT 

11 SAL RT SUB LIFEBOATS AND LIFERAFTS 

12 SAL NT SUB LIFEGUARDS 

Sa6 accident victim names 

Sa6 personal names 

Sa6 sport names 

* * *END OF DISPLAY* * * 



As an extension of the above discussion on automatic cross-reference Sf 
another featxire of the Information Bcink must be noted. When there is a single 
synonym for a term in the thesaurus there is an automatic switch from the form 
entered by the user to the legal version. For example, if a user were to enter 
the term 'Vo^th", the system would automatically switch to the accepted file of 
"children and youth". 

The normal procedure for automatic switching is sixrh that the user does 
not become aware of it until he reaches the logic stage. Of course, if the user 
is doing a search requiring only one index term, he will never see the term the 
system has picked for him. Since switching also takes place from a narrow to a 
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broad term, the user may be surprised to find that by entering a single narrow 
term (such as 'Tiours of labor") he may ''exceed the system limit," because his 
inquiry is switched to the general file on "labor". 

As the Information Bank has been adding to its automatic switches and 
cross-references, the unassisted user has had to face the confusion without 
any aid from the system. It seems only fair to emphasize the existence of 
such features to the novice user either through on-line instructional aids 
or easily accessible written guidelines. 

From an instructional standpoint, a sample cross-reference screen 
reveals the following problems: 

1. Many abbreviations are used which are not explained on-line, 
promoting the mystique of "computerese . " The most common corment at the 
cross-reference step is "I don't understand what it is trying to tell me." 
With very little effort or loss of space, certain of these abbreviations may 
be converted to natural language. For example, it takes only one extra char- 
acter to turn a meaningless "NTE" into its English equivalent, "NOTE". 

When a user is attempting a first search on the Information Bank there 
is enough novelty in the system and the procedures to occupy his mind. He must 
not be burdened with extra effort to decipher unfamiliar abbreviations. 

2. Although the "B" and "D" options seen to allow the user to go to 
the next step, what actually happens is that he is taken back to the previous 
step. For example, the user may be viewing terms on a term selection screen; 
he asks to view the cross-reference to one of the terms; he then proceeds to 
the "next step" from the cross-reference screen and finds himself back at the 
original term selection screen he had viewed earlier. This is a very helpful 
and essential feature, but rather confusing to the new user. Perhaps revising 
the wording of the "A" and "D" commands can alleviate the problem for the novice* 
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J. Modification Stage: First Step 



If you wish to restrict inquiry by other search elements: 

A. View term modifiers (byline, sketch, weight), ENTER: A 

B. View other modifiers (date, source, etc.), ENTER: B 

C. View both term and other modifiers , ENTER: C 

D. No restrictions — go to next step, ENTER: D 

An obvious question at this step is what are "other" modifiers? In 
the abbreviated forms of system messages the seune option is referred to as 
"bibliographic modifiers!'. A simple suggestion would be to replace the term 
••other" with ••bibliographic" in the full forms of system messages. 

Extending the function of the universal ''W^^ key throughout the modi- 
fication stage can be of great help in explaining the meaning of various 
modifiers. 

The ••D^^ option can also benefit from a slight modification. Following 
the established pattern of system messages, this option may be revised to read 
D. No restrictions (save prior entry). Go to next step, ENTER: D 
This would save the novice user the extra steps involved in re-entering a modi 
fication he may have established during an earlier portion of his search. At 
present the system does not inform the user in any way that all modifications 
entered during a search remain intact even when new terms are added to the 
search. 

From the academic user's point of view the ••B" option is used much 
more heavily than •'A". If the majority of the system's users confirm this 
preference it may be more efficient to switch the ••A'* and ''B" options at this 
stage. 
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K. Modification Stage: Term Modification 



Term modifiers are denoted by abbrevs as follows: 
byline==ABT (about) or BYL (by) 
sketcb^DRF (brief) or DTL (detailed) 
term weight (least to greatest) =1^ 2, 3 or 4 

A. Apply abbrev to 1 or more search terms 

ENTER: A/ /term nxmber ^abbrev/ / term nmtber <abbrev/ / . . .etc. 

B. View continuation f ENTER: B 

C. Go to the next step, ENTER: C 



'As mentioned in the previous section, this option does not receive 
much use in this academic environment. However, the basic misunderstandings 
which occur are related to definition of terms rather than following the 
instructions. 

TO an academic user a "sketch" may mean an "outline" or a "drawing". 
Jfc one related this term to biographies. It would certainly be to the advan- 
tage of the non- journalist user to get a better understanding of this term 
on-line, preferably through an expansion of the instructional message. The 
sme procedure may also be used to define "about line" and "term weight" to 
the uninitiated user. 
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Modification Stage: Other Modifiers 



Select other means to modify search from this list 

1. date 4. type of material 7. section 

2. journal 5 . illustrations 8. page 

3. source 6. abstract weight 9. column 

A. Select 1 or more modifiers 

ENTER: A/ /item number //item number//. . .etc. 

B. Skip this, go to next step, ENTER: B 

Although the instructional message is very simple to follow, the 
problem of definition of terms also applies to this step in the modification 
process. 

"Source" is not easily interpreted as "press agency" by this non- 
Journalist user group. In fact, most users interpret "source" from a 
library standpoint, thus equating it with the name of the journals. 

"Type of material" has no meaning to the novice user, but with the 
exception of "date", it is the most often used option once the meaning has 
been made clear to the user. At the modification stage there is no screen 
display beyond the instructional messages. It would seem reasonable to ex- 
pand some portions of the message to provide clarification of individual 
terms used. 
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M. Modification Stage: Date Modification 

Date is year-month-day Example 1971^12-31 

A. Limit material to 1 date of publication, ENTER: A//date 

B. Limit to up to 8 alternate dates, ENTER: B/ /date/ /date/ /etc. 

C. Limit to 1 date range, ENTER: C//date TO date 

D. Limit to up to 8 ranges, ENTER: D//date TO date//. . .//etc. 

E. Limit to dates/ ranges , ENTER: E//date//. . .//date TO date/ /etc. 

F. Skip this (or cancel prior entry); go to next step, ENTER: F 

G. Save prior entry if any; go to next step, ENTER: G 

This is by far the most popular mode of modification in this academic 
community. The instructions are very simple and easy to understand. Only one 
comment need be made here. Ihe date example offered on-line misleads the user 
into thinking that he must always include year, month, and day in his request. 
Ihus, when he wishes to limit his query to a given year, say 1973, he chooses 
the "C" option and responds as follows: 

C//1-1-1973 to 12-31-1973 

instead of the much simpler option available to him: • 

A//1973 

Perhaps the date example can be expanded to include this time-saving 

hint. 

In Section I of this chapter it was suggested that the user be given 
the opportunity to check his last response. This is one of the stages where 
many users wish to confirm a modification they have just entered; perhaps to 
double check on its accuracy r Although "term modifiers" visibly tag the 
affected term at the logic stage, no indication of bibliographic modification 
is present until the user begins to view abstracts. It would be helpful to 
have an early indication of those modifiers accepted by the system. 
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N« Logic 



A. Link terms with logical connectives and view remaining terms. 
ENTER: A//tQrm number and (or^ not) term number and. . .etc. 

B. Link terms with logical connectives and begin search. 
ENTER: B//term number and (or^ not) term number and. . .etc. 

C. View continuationf ENTER: C 

Here, as with many of the previous steps discussed, the novice user 
chooses the "A" rather than the "B" option not because It Is applicable to 
his search, but rather because It was the first instructional message on the 
screen. Ihe majority of searches in the academic community are carried out 
without using the paging option available. Again, If this proves to be the 
case with other user groups as well. It may be worthwhile to consider placing 
the most-used option at the top of the screen. 

It Is also noteworthy that the present instructions imply that a link- 
ing of terTHS is necessary in every case. A user who may have picked three 
terms, one with only a single citation, may wish to view that abstract alone. 
The instructions do^ not guide him as to how he may proceed with such a request. 

Because of an Infinite variety of errors possible at this stage, many 
of the present error messages tend to mislead the user. The following is a 
case in point. A user had selected the terms "Confuclous" and ''cultural 
revolution." At the logic stage he typed the following message: 

b//confuclous and cultural revolution 

The system responded with the following error message: 

"an OR AND or NOT did not follow a # or right paren" 



122 

134 



if the user had been conceptually aware of the procedure for linking terms i he 
may have been able to decipher the error message, but in this case the error 
analysis provided no relief • 

Other cases of errors involve the concept of "Boolean operators It 
is suggested that either Boolean errors be diagnosed in more detail to offer 
relevant error messages, or to simply revert to the system's standard error 
messages, and allow the user to re-evaluate his own response. Again, it 
would be helpful to the user if he could see his last res{X>nse at the time 
he received the error message. In any event, a helpful addition to this stage 
would be to offer the user, upon demand or after two consecutive failures to 
formulate correct logical connectives, some simple hints or lessons on Boolean 
logic* 

For those users who choose the "A" option from the logic stage, the 
system responds with this message: 

A* Replace logic and view continuation. 

ENTER: A//term number and (or, not) term number and . . .etc, 
B. Replace logic and begin search. 

ENTER: B//term number and (or, not) term number and . . .etc. 

View continuation , ENTER: C 

D. Add to logic and view continuation. 

ENTER: D//term number and (or, not) term number and . . .etc. 

E. Add to logic and begin search. 

Enter: E//term number and (or, not) term number and . . .etc. 

F. Logic complete, begin search, ENTER: F 

Most novice users do not attempt searches which require paging. 
Ihus, the difficulties in performing a search of this type will not be dis- 
cussed in this report. However, two points must be made here. First, the 
"C" option to •View continuation*' is missing from the screen. John Rothman 
has pointed out that this was due to an oversight and will be corrected. 
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Another Item which Is exclusive to this screen is that the user's res 
ponse. is carried over from the previous page. Thus we are assured that the 
capability of providing such information is within the present bounds of the 
system. As suggested before, it would be helpful to expand this provision to 
other steps of the search. 

III. An Overview 

One of the outcomes of the Stanford workshop of on-line systems 
designers was the consensus reached on a number of features deemed to be 
"essential" for interactive searching. These minimal features are as follows 

1. User's guide; preferably complete and usable 

2. Live help; by telephone or message-command 

3. Suffix removal 

4. Search field control 

5. Relational operators for numeric data bases 

6. Boolean operators 

7. Request sets; building the logic upon previous sets 

8. Search review; allowing the user to retrace his steps 

9. Pre-defined formats; including fields, short citations 
or full documents 

10. On-line formatting; for the user with special needs 

11. Off-line printing (10) 

Of these eleven features, only four are fully available in The New York Times 
Information Bank. These are the user guides, live help, suffix removal by 
truncation, and Boolean logic operators. 

Ihe relational operators are not necessary for this particular data 
base. Ample provisions are made for all numeric fields (such as date, page 
and section nvjunbers) used in the data base. 
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Independent search field controls are .not available. All searching 
must initiate through the use of index terms» Itiat is, one cannot aslc for the 
front page news of a particular day, or all the by-lines of a given week; nor 
can a search be conducted by titles of articles, or through a specified set of 
abstracts. Although none of these options may appear useful to the newspaper 
conminity, some can be of great help to those in an academic environment. 

^ Request sets and search review options would also be of assistance to 

the user who must currently rely upon his memory to re-construct the previous 
segments of his search. It would be especially useful at the logic stage to 
be able to review previous logical connectives made and their respective 
search results. 

Pre-defined formats and on-line formatting options have been suggested 
as useful additions to the system by many users. It is considered as one of 
the major shortcomings of the present system that only a single, pre-defined, 
format for output is available. Ihe user must vie\v both citation and abstract 
for each item* The simplest option mo^t frequently requested by users is a 
"citation only" output. Those users who fully intend to refer to the full 
text of articles retrieved consider it a waste of time and money to obtain 
prints of abstracts when the citation alone meets their needs. 

Off-line printing is also not currently available to subscribers. 
This is a great limitation to those doing extended research on large files^ 
Ihe only option at present is to print citations and abstracts one page at a 
time. As this time-consuming procedure also ties up the terminal for the 
duration of the printing process, the cost for prolonged printing becomes 
prohibitive. 
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An even more obvious waste which could be eliminated is the rep>eated 
printing of instructional messages which appear at the top of each screen. 
It seems that the CRT program could be modified to block the hard-copy print- 
ing of the messages at this stage. 

It has not been the intention of this author to burden a single system 
with all features deemed useful or necessary by various designers or user 
groups. On the contrary, there sure many additional features not mentioned in 
this report, but currently available in other systems which could be adapted 
by The New York Times Information Bank to the benefit of all its users. A 
single relevant example will be cited. When a user arrives with a clipping 
of an article — or finds a single article on-line — he may comment that he 
needs other articles on the same topic. It would be extremely helpful if 
the Information Bank provided the index terms attached to a given 
abstract. Other examples abound, but the author has specifically avoided 
discussion of features which seem to fall outside the objectives or scope 
of services provided by the Information Bank. It is hoped that the analysis 
offered will be accepted on that basis and open the way to future changes 
which would reflect the needs of all non-specialist users of the system. 
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SUMMARY AND CONCLUSIONS 



This study was undertaken to test the effectiveness of a computer- 
assisted tutorial program in teaching the novice user the procedures for 
conducting a search on The New York Times Information Bank. It was also 
hoped that in the course of the experiment the learning process of the 
novice user would be traced to Isolate the thesaural from mechanical prob- 
lems. As a very important by-product, the process of this study was to yield 
a detailed analysis of search failures due to misinterpretation of on-line 
instructional and error messages. 

The quantitative as well as the qualitative analysis of data offered 
in chapters 5 and 6 clearly support the first hypothesis related to the 
• effectiveness of the tutorial program. However, because of the developments 
in the course of the experiment which revealed that the tutorial group did 
not refer to the printed instructions offered as a part of the system, this 
hypothesis may be amended and accepted as follows: 

Ihe computer-assisted tutorial program for The New York Times Informa- 
tion Bank is a more effective tool than the printed instructions offered by 
the system for learning to use the Information Bank. 

What is also clearly shown by the data is that the tutorial program, 
though more effective than the printed instructions, is far from being a fully 
effective training medium. 
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Some of the shortcomings of this particular CAl program were detailed 
in chapter 7, but it would also be wise to consider the inherent shortcomings 
of this or any other single medium. The Intrex project showed that a combina**" 
tion of several training methods worked better than any single mode used alone. 
It also brought to light the fact that even with such combination all users 
could not be satisfied at all times, il) The Stanford Workshop revealed that a 
majority of on-line system designers feel that a combination of live help and 
printed instructions are essential ingredients of an interactive bibliographic 
retrieval system. (2^) On the other hand, although we find that all present 
systems offer some form of printed instructions as well as live help, this 
combination has also failed to provide the desired results. 

Davis hfcCam of the National Library of Medicine, in a study con- 
diicted in March, 1973, encountered the following reasons for the failure of 
live training of the medical community to use MEDLINE. 

1. The staff's lack of ease in dealing with the novice user. 

2. Inexperience in teaching. 

3. Preservation of the status symbol for the instructor (who 
may feel that he may no longer be of service if the ultimate 
user becomes fully proficient in handling the system. ) 

4. The instructor's low estimate of patron capability. 

5. The instructor's personal interpretation of the "proper" use 
of the system. 

6. The National Library of Medicine's establishment of a fee 
schedule based upon connect time which discourages unre- 
stricted use of the system. (3^) 

We will find that the last item applies to all commercial and profit / 

making systems as well. The New York Times allows two months of unrestricted 

use, but depending on the individual contract, connect' time changes apply after 
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the second month. Ihe remaining five causes, however, are all related to the 
biases of live help . Thus, although it remains the author* s strong belief 
that live help is presently essential for acquainting novice users with an 
on-line system, it is deemed just as essential to provide the user with alter- 
native means of obtaining the desired information. An on-line tutorial pro- 
gram, built as an integral part of the information retrieval system is seen as 
the ideal alternative or back-up to live help. 

The second hypothesis of this study sought to separate the learning 
process at the terminal into its '•mechanical" and "thesaural" elements. The 
qualitative results of the experiment reveal a refinement of this hypothesis. 
It was shown that in the order of most to least importance, the user may en- 
counter one or more of the following problems: conceptual, interpretational, 
thesaural, and mechanical. Such problems could be easily isolated in some 
cases, but usually proved to work in a chain reaction. The highest degree of 
failures could be ultimately attributed to the conceptual misinterpretation 
of the function of the system in general, or the flow of steps making up the 
search process in particular. This was followed closely by problems related 
to the interpretation of instructional or error messages appearing on-line. 

The present study did not reveal any significant difference in results 
between the sections receiving assistance in term selection and those left to 
conduct their own term negotiation. Unfortunately, the small sample size in 
Section II (24 participants) does not permit the author to either confirm or 
deny thesaural problems as a major cause for search failures. However, a more 
detailed look at individual "term selection" processes revealed that many mis- 
understandings resulted from the physical layout of the term selection screen 
and the abbreviations used on-line in connection with thesaural displays. 
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Mechanical problems faced by the participants did not lead directly to 
search failure • They were usually an indication of another, more important i 
problem which the user was facing at the time. However, many pure mechanical 
problems— such as using the Ic^er case letter "Z" for the numeral "1" and other 
typographical errors — are analyzed incorrectly by the system, thus misleading 
the user. Being unaware of the actual cause of error, the user may fail to 
complete a search due to mis-interpretation of instructional messages. Chapter 
8 of this study has dealt in detail with problems related to the instructional 
messages appearing on-line. 

The second hypothesis may thus be accepted with the following refine- 
ments : 

The learning process at the terminal is divided into three separable 
areas: The concept of formulating a search strategy, mastering the use of 
the on-line thesaurus of the Information Bank, and correctly interpreting the 
on-line instructional messages. 

In looking back at the questions posed prior to the conduct of the 
study (chapter 3), we find that this experiment has provided some insight 
into the novice user's learning process. It has shown some of the advantages 
of the CAI program over printed instructions; and proved that familiarity with 
the mere mechanics of formulating a search strategy does not guarantee a suc- 
cessful search. Although we have in fact seen that certain individuals do not 
respond to printed or computer-assisted instructions, no attempt was made to 
find a suitable medium for all users of the system, or to prove that certain 
individuals may be characteristically incapable of using an on-line informa- 
tion retrieval system- with complete ease and success. 

Recommendations for Future Study 

1. Because of this study's failure to delve deeper into the problems 

related to the on-line thesaurus of - Ihe New York Times Information Bank, it 
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is suggested that a follow-up study concentrate on this aspect of the novice 
user*s search problems. 

A tutorial program dealing with the problems of term selection and 
entry has been developed for this purpose. It may be useful to test its 
effectiveness in a similarly designed experiment. 

2. It would be of interest to compare the results obtained from the 
present study with a control group receiving live instructions of similar 
duration. As an extension of the comparison of various training media, it 
would also be worthwhile to test the effectiveness of a slide-show produced 
by the Campus-Based Information System (using the same sample question as the 
CAI program). Since the instructional content of all sessions can be kept 
constant, such a study may provide a particular direction to future efforts 
expended in the area of user training. 

3. The present study was conducted in an academic environment. If 
the results obtained in this setting correspond with those obtained from other 
segments of the user population, it would certainly add strength to the recom- 
mendations for modifying some of the on-line instructional messages. On the 
other hand, if it were found that such results are unique to the academic com- 
munity, it would be an indication to The New York Times that their system is 
more suited to a particular segment of the universe of users. Such indication 
may lead to a redirection of marketing efforts on the on^e hand, or design 
efforts on the other, to adapt the system to its ultimate user group. It is 
thus strongly lurged that similar studies be conducted with other segments of 
the user population. 

4. It is essential that the students emd designers of on-line, inter- 
active bibliographic retrieval- systems look upon the human side of man/machine 
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interaction as one of the major ingredients of such a system. Ihomas Martin's 

comments on this topic may be considered the basis for the broadest spectrum of 

research which needs to be done in this area: 

The manner in v/hich a person interacts with a compniter- 
ized system and the manner in which he carries out his 
searching tas)c are not isolable phenomena. . They are 
instances of behavior and can best be understood in a 
broader corrjnunication and/or problem-solving context. 
One can only discover what is unique about man-computer 
communication and interactive searching by contrasting 
each with its respective behavioral background. While 
principles can be derived from experience, there is a 
need to relate the principles to similar phenomena and 
to carry out tests that measure the reliability of the 
principles. U) 
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